Senior Software Engineer, AI Data

NVIDIA•Durham, NC

1d•$152,000 - $287,500

About The Position

NVIDIA’s Managed AI Research Superclusters (MARS) team builds and scales infrastructure, platforms, and tools for developing AI/ML systems. The MARS AI Data team manages the data layer used by NVIDIA AI research, including petabytes of training data, checkpoints, and datasets across multiple clouds and partners. The team builds the layer between storage and workloads, covering data placement across file systems, object storage, and local NVMe; streaming loaders and caches for large GPU fleets; checkpoint management; and fair quotas across research groups. In this role, you will help design MARS’ data platform on Kubernetes from the ground up and work directly with researchers training NVIDIA models. Your work will support faster training, shorter restarts, and fewer runs lost to storage issues. Responsibilities span data placement and movement, high-throughput data paths, Kubernetes-native platform integrations, quotas and lifecycle management, I/O telemetry and analytics, and reliability ownership.

Requirements

Bachelor’s or master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
At least 5 years building production systems or data-infrastructure software.
Deep experience with at least one of: parallel or distributed file systems such as Lustre, WEKA, VAST, GPFS, or Ceph; object storage at scale such as S3 or compatible systems; or large-scale data pipelines and data lakes.
Clear understanding of I/O performance, including throughput, latency, metadata cost, caching, and how access patterns interact with them.
Strong programming skills in Go, Rust, C, or C++, plus Python for tooling and analysis.
Track record of owning a production service and measurably improving it.
Clear written and spoken communication with engineers and researchers.

Nice To Haves

Experience feeding large-scale AI training workloads, including data loaders, streaming datasets, and storage behavior at thousands of GPUs.
Experience with checkpointing at scale, including asynchronous or distributed checkpoints, tiered checkpoint storage, and fast restart after failure.
Experience building or operating a distributed cache or AI-oriented storage system, or using node-local NVMe as a managed tier.
Experience building a quota, accounting, or capacity system used by many teams.
Knowledge of reinforcement learning and agentic workloads and their storage needs, including many small reads, environment snapshots, sandbox images, and fast fan-out.

Responsibilities

Design and build the AI data layer for a multi-cloud GPU fleet, including placement, caching, tiering, and movement of datasets, checkpoints, and scratch data across storage backends.
Build high-throughput data paths for training and reinforcement learning, including object-storage streaming loaders, distributed caches, and checkpoint save and restore at thousands of GPUs.
Implement peer-to-peer and local-first checkpoint restore capabilities.
Build Kubernetes-native APIs, operators, and storage integrations that let jobs declare their data needs and allow the platform to mount, cache, and account for data on Slurm and Kubernetes.
Own quotas, capacity, and data lifecycle across research teams, including usage attribution, quota enforcement, retention, stale-data cleanup, and capacity forecasting.
Build I/O telemetry and access analytics to identify hot and cold data, metadata pressure, and I/O stalls, and use findings to guide data placement and fixes.
Own reliability for the systems built, including on-call, incident analysis, and changes that prevent recurring data incidents such as full file systems, runaway metadata load, and lost checkpoints.