Senior Software Engineer, AI Data

NVIDIA•Santa Clara, CA

About The Position

NVIDIA’s Managed AI Research Superclusters (MARS) team builds infrastructure, platforms, and tools for AI/ML research and engineering. The MARS AI Data team manages the data layer for NVIDIA AI research, including petabytes of training data, checkpoints, and datasets across multiple clouds and partners. It builds the layer between storage and workloads, covering data placement across file systems, object storage, and local NVMe; streaming loaders and caches for large GPU fleets; checkpoint management; and fair quotas across research groups. In this builder role, you will help design MARS’ data platform on Kubernetes from the ground up and work directly with researchers training NVIDIA models. The work aims to improve training speed and restart times while preventing lost runs caused by full storage volumes. You will build data infrastructure for a multi-cloud GPU fleet, make data a first-class part of the compute platform, manage usage and lifecycle across research teams, measure data access and I/O behavior, and own reliability for the systems you build.

Requirements

Bachelor’s or master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
At least 5 years of experience building production systems or data-infrastructure software.
Deep experience with at least one of the following: parallel or distributed file systems such as Lustre, WEKA, VAST, GPFS, or Ceph; object storage at scale such as S3 or compatible systems; or large-scale data pipelines and data lakes.
Clear understanding of I/O performance, including throughput, latency, metadata cost, caching, and how access patterns interact with these factors.
Strong programming skills in Go, Rust, C, or C++, and Python for tooling and analysis.
Track record of owning a production service and measurably improving it.
Clear written and spoken communication skills with engineers and researchers.

Nice To Haves

Experience feeding large-scale AI training workloads, including data loaders, streaming datasets, and storage behavior at thousands of GPUs.
Experience with checkpointing at scale, including asynchronous or distributed checkpoints, tiered checkpoint storage, and fast restart after failure.
Experience building or operating a distributed cache or AI-oriented storage system, or using node-local NVMe as a managed tier.
Experience building a quota, accounting, or capacity system used by many teams.
Knowledge of reinforcement learning and agentic workloads and their storage needs, including many small reads, environment snapshots, sandbox images, and fast fan-out.

Responsibilities

Design and build the AI data layer for a multi-cloud GPU fleet, including placement, caching, tiering, and movement of datasets, checkpoints, and scratch data across storage backends.
Build high-throughput data paths for training and reinforcement learning, including streaming loaders from object storage, distributed caches, and checkpoint save and restore across thousands of GPUs.
Develop peer-to-peer and local-first checkpoint restore capabilities.
Build Kubernetes-native APIs, operators, and storage integrations so jobs can declare data needs and the platform can mount, cache, and account for data on Slurm and Kubernetes.
Own quotas, capacity, and data lifecycle across research teams, including usage attribution, quota enforcement, retention, stale-data cleanup, and capacity forecasting.
Build I/O telemetry and access analytics to identify hot and cold data, metadata pressure, and I/O stalls, and use those insights to guide placement and fixes.
Own reliability for the systems developed, including on-call, incident analysis, and changes that prevent classes of data incidents such as full file systems, runaway metadata load, and lost checkpoints.