HPC Storage Engineer - West Coast

Runpod

2d•$180,000 - $260,000•Remote

About The Position

Runpod is the AI Developer Cloud, used by over a million developers to experiment, train, fine-tune, deploy, and scale AI. The platform has processed over 20 billion inference requests. We are seeking a Senior Storage Engineer to join our Infrastructure organization, specifically within the team managing Runpod's multi-region storage ecosystem, including network volumes, local NVMe, and S3-compatible object storage. This is a hands-on engineering role focused on the design, scaling, and reliability of Runpod's storage platform. You will implement and maintain distributed storage deployments, write code and automation, and contribute to strategic decisions about storage systems, data tiering, placement, network tuning, and petabyte-scale hardware acquisition. You will have significant autonomy to innovate, automate manual operations, define metrics and SLOs, and lead capacity expansions and migrations. Improvements made by this role will directly impact customer experience through faster cold starts, quicker jobs, and reduced incidents for over a million developers.

Requirements

8+ years in infrastructure, storage, or systems engineering, with substantial ownership of production storage at scale.
Deep, practical experience with at least one distributed storage system — Ceph, MinIO, Lustre, GPFS/Spectrum Scale, MooseFS, WekaFS, VAST, ZFS-based systems, or comparable.
Strong Linux internals and storage-stack knowledge: block layer, filesystems, NVMe, page cache, I/O schedulers, NFS/SMB, iSCSI/NVMe-oF.
Building and/or operating S3-compatible object storage services.
Solid networking fundamentals with specific experience tuning networks for storage workloads.
Proficiency in writing and shipping production code in Go, Python, Rust, or similar (not just scripting).
Hands-on experience with observability tooling (Prometheus, Grafana, Datadog, or equivalent) including designing the metrics, not just consuming them.
A track record of performance analysis and debugging under real production pressure.
Self-starting with general direction. You take a goal like "network volume read latency is hurting cold starts in EU" and come back with a diagnosis, an options analysis, and a plan without needing the work broken down for you.
Continuous improvement. You leave systems measurably better than you found them. You notice the recurring toil, the alert that fires every Tuesday, the manual step everyone tolerates and you eliminate it.
Ownership. You follow problems across team boundaries to resolution instead of handing them off at the edge of your component.
Collaborative and low-ego (but high confidence).

Nice To Haves

Storage for AI/ML workloads: checkpointing, dataset streaming, model weight distribution, GPU-adjacent data locality, GPUDirect Storage.
Kubernetes storage internals: CSI drivers, PV/PVC lifecycle, StatefulSets, local persistent volumes.
Bare-metal and colocation experience: hardware selection, vendor management, firmware, physical failure domains.
Multi-tenant environments where isolation, fairness, and QoS are hard requirements.
Experience in a fast-growing cloud or infrastructure provider.

Responsibilities

Own capacity, durability, availability, and performance characteristics of network volumes, local NVMe, and S3-compatible object storage.
Tune the full I/O path: device and filesystem configuration, caching and read-ahead strategies, replication and erasure coding trade-offs, and client-side mount behavior.
Diagnose hard performance problems end to end.
Lead capacity expansions, hardware refreshes, migrations, and rebalances without customer-visible disruption.
Work with Runpod and partner networking teams to design and tune the network paths storage depends on: high-throughput east-west fabric, MTU and jumbo frames, congestion and flow control, multipath, and NIC/offload configuration.
Understand and optimize RDMA/RoCE and high-speed IB/Ethernet fabrics as they apply to storage traffic.
Work closely with network engineering on topology decisions, oversubscription ratios, and cross-region data movement.
Write production code (Go, Python, or similar) for storage control-plane services, provisioning workflows, data movement pipelines, and monitoring.
Build against and extend APIs: our own control plane, S3-compatible interfaces, CSI drivers, Kubernetes APIs, vendor and cloud provider APIs.
Automate the operations you'd otherwise do by hand. Manual runbooks are a starting point, not a destination.
Treat infrastructure as code and participate fully in code review, testing, and CI.
Instrument the storage fleet so its behavior is legible: IOPS, throughput, latency, error and retry rates, capacity utilization, and per-tenant consumption.
Build dashboards, SLOs, and alerts that catch degradation before customers do.
Participate in an on-call rotation for storage systems and drive blameless post-incident follow-through.