Cross-Layer Data-Integrity Qualification of GPU-Direct Storage Paths Using NVMe-oF and DPU Offload
DOI:
https://doi.org/10.71238/snnst.v2i02.201Abstract
GPUDirect Storage over NVMe over Fabrics (NVMe-oF), more often now combined with data processing unit (DPU) offload, has reduced the latency from flash to accelerator memory but eliminated the host-side copy at which payloads were conventionally sifted, and silent data corruption (SDC) is a documented problem at fleet scale. It was desired to find a cross-layer data-integrity qualification scheme for these paths. We performed a systematic review of 17 publications (3 journal articles, 7 conference or book-series papers and 7 preprints). The evidence was structured with nine equations that describe residual corruption probability, digest coverage, verification overhead, offload overlap and test-pass sizing and was categorised by function (path layer) and offloading. There were five path layers related to fault classes, detection mechanisms, qualification tests. NVMe-oF was reported to increase the access time by 11.7 µs, while iSCSI decreased key-value throughput by 40%. Offload gains of 15% in training time and 2.8 times in throughput were achieved without reporting any overhead for verification. In an illustrative model, transport-only digests left about 977 expected undetected events per exabyte, while end-to-end coverage left 8.0 × 10−7 expected undetected events per exabyte. The level of integrity is a path-level property and needs to be accompanied by overheads for integrity verification along with offload efficiency. The review provides a multi-layer qualification matrix and a quantitative model that are still valid in case of migrating storage functions to the DPU.
Downloads
References
I. Burstein, “NVIDIA data center processing unit (DPU) architecture,” in 2021 IEEE Hot Chips 33 Symposium (HCS), IEEE, 2021, pp. 1–20. doi: 10.1109/HCS52781.2021.9567066.
H. D. Dixit, L. Boyle, G. Vunnam, S. Pendharkar, M. Beadon, and S. Sankar, “Detecting silent data corruptions in the wild,” 2022. doi: 10.48550/arXiv.2203.08989.
Z. Guz, H. Li, A. Shayesteh, and V. Balakrishnan, “Performance characterization of NVMe-over-Fabrics storage disaggregation,” ACM Trans. Storage, vol. 14, no. 4, Art. no. 31, 2018, doi: 10.1145/3239563.
D. Inupakutika, B. Davis, Q. Yang, D. Kim, and D. Akopian, “Quantifying performance gains of GPUDirect Storage,” in 2022 IEEE International Conference on Networking, Architecture and Storage (NAS), IEEE, 2022. doi: 10.1109/NAS55553.2022.9925516.
A. Jain, N. Alnaasan, A. Shafi, H. Subramoni, and D. K. Panda, “Accelerating CPU-based distributed DNN training on modern HPC clusters using BlueField-2 DPUs,” in 2021 IEEE Symposium on High-Performance Interconnects (HOTI), IEEE, 2021, pp. 17–24. doi: 10.1109/HOTI52880.2021.00017.
A. Kashyap, S. Gugnani, and X. Lu, “Impact of commodity networks on storage disaggregation with NVMe-oF,” in Benchmarking, Measuring, and Optimizing, F. Wolf and W. Gao, Eds., in Lecture Notes in Computer Science, vol. 12614. Springer, 2021, pp. 41–56. doi: 10.1007/978-3-030-71058-3_3.
H. D. Dixit et al., “Silent data corruptions at scale,” 2021. doi: 10.48550/arXiv.2102.11245.
E. Kfoury, S. Choueiri, A. Mazloum, A. AlSabeh, J. Gomez, and J. Crichigno, “A comprehensive survey on SmartNICs: Architectures, development models, applications, and research directions,” 2024. doi: 10.48550/arXiv.2405.09499.
K. Kourtis, A. Trivedi, and N. Ioannou, “Safe and efficient remote application code execution on disaggregated NVM storage with eBPF,” 2020. doi: 10.48550/arXiv.2002.11528.
A. Li et al., “Evaluating modern GPU interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect,” 2019. doi: 10.48550/arXiv.1903.04611.
P. MacArthur, Q. Liu, R. D. Russell, F. Mizero, M. Veeraraghavan, and J. M. Dennis, “An integrated tutorial on InfiniBand, verbs, and MPI,” IEEE Commun. Surv. Tutorials, vol. 19, no. 4, pp. 2894–2926, 2017, doi: 10.1109/COMST.2017.2746083.
B. Michalowicz, K. K. Suresh, H. Subramoni, D. K. Panda, and S. Poole, “DPU-Bench: A micro-benchmark suite to measure offload efficiency of SmartNICs,” in Practice and Experience in Advanced Research Computing (PEARC ’23), ACM, 2023, pp. 94–101. doi: 10.1145/3569951.3593595.
N. Sarkauskas, M. Bayatpour, T. Tran, B. Ramesh, H. Subramoni, and D. K. Panda, “Large-message nonblocking MPI_Iallgather and MPI_Ibcast offload via BlueField-2 DPU,” in 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC), IEEE, 2021, pp. 388–393. doi: 10.1109/HiPC53243.2021.00054.
M. Silberstein, B. Ford, I. Keidar, and E. Witchel, “GPUfs: Integrating a file system with GPUs,” ACM Trans. Comput. Syst., vol. 32, no. 1, Art. no. 1, 2014, doi: 10.1145/2553081.
D. Unat et al., “The landscape of GPU-centric communication,” 2024. doi: 10.48550/arXiv.2409.09874.
Z. Yang, Q. Wan, G. Cao, and K. Latecki, “uNVMe-TCP: A user space approach to optimizing NVMe over fabrics TCP transport,” in Internet of Vehicles. Technologies and Services Toward Smart Cities (IOV 2019), C.-H. Hsu, S. Kallel, K.-C. Lan, and Z. Zheng, Eds., in Lecture Notes in Computer Science, vol. 11894. Springer, 2020, pp. 125–142. doi: 10.1007/978-3-030-38651-1_13.
I. Zarkadas et al., “BPF-oF: Storage function pushdown over the network,” 2023. doi: 10.48550/arXiv.2312.06808.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Gagan Vyas (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Facebook 
Instagram 
Twitter 
Youtube 
+62 822 2101 3235
northpressmail@gmail.com