Senior Storage Engineer

Runware • Palo Alto, CA

2d • Remote

About The Position

Runware is building an AI Factory to operate its Sonic Inference Engine, which serves more than 20 billion requests and supports more than 400,000 models. As Senior Storage Engineer, you will own the storage architecture for Runware’s AI supercomputer, designing how model weights, caches, and customer data move from object storage across two continents to NVMe and GPU memory throughout the fleet. This is a hands-on senior individual-contributor role: you will design the data path, implement its first version, benchmark systems on real hardware, diagnose performance issues, and establish operational standards for storage across every pod.

You will work with inference engine, platform, and hardware teams; storage and NVMe vendors; and site teams deploying pods. The goal is to load cold models onto GPUs anywhere in the fleet within seconds, warm new pods before they receive production traffic, and make storage failures at the disk, node, or site level invisible to API developers. The role also owns storage reliability, capacity planning, security, data residency, and operational readiness. Runware is remote-first, with core hours for collaboration, in-person meetings twice a year, and travel to US and European sites as needed.

Requirements

• At least 8 years of experience in storage or systems engineering, including several years designing and operating storage at a company that runs its own hardware at scale.
• A track record of building storage systems from design through petabyte-scale production, rather than only operating systems built by others.
• Deep, current, hands-on Linux storage expertise, including the block layer, io_uring, NVMe queues, file systems such as XFS, ext4, and ZFS, the page cache, and I/O diagnostics.
• Production experience with at least one parallel or distributed file system, such as Lustre, WEKA, VAST, GPFS, or Ceph, and with S3-compatible object storage at petabyte scale.
• Strong understanding of NVMe, NVMe-oF, RDMA transports, and storage networking, including the ability to diagnose bottlenecks involving PCIe links, drives, and switches.
• Fluency in a systems language such as Go, Rust, or C/C++, and Python; ability to write tooling, benchmarks, and automation.
• Regular use of AI to accelerate scripting, benchmarking, analysis, documentation, and debugging, and a clear view of how these tools can change the way a small team operates.
• Ability to own outcomes in a fast-moving, ambiguous environment, make decisions with incomplete information, and be accountable for latency, durability, and cost.
• Ability to travel to sites in the US and Europe as needed.

Nice To Haves

• Experience feeding large GPU fleets for AI training or inference, including GPUDirect Storage, NCCL-aware data loading, model weight caching, or loaders in vLLM, SGLang, TensorRT-LLM, or similar engines.
• Hands-on experience with InfiniBand or RoCE fabrics and 400G-or-faster Ethernet from the storage side.
• Experience with content-addressed or deduplicated storage, erasure coding, and multi-region replication and consistency trade-offs.
• Kernel or driver contributions, or deep familiarity with SPDK, DPDK, or user-space storage stacks.
• Experience with Kubernetes CSI, Ceph or Rook on bare metal, and automation such as Ansible and Terraform with Prometheus and Grafana observability.
• Familiarity with the storage hardware supply chain, including drive qualification, firmware lifecycle, enclosure and backplane design, spares, and RMA economics.
• Experience building or running storage to meet data residency, sovereignty, or compliance requirements across jurisdictions.

Responsibilities

• Own the fleet’s end-to-end storage architecture, including durable object storage, per-pod high-throughput storage, node-local NVMe caching, replication, and placement across the US and Europe.
• Design and build the model distribution path, including how weights are transferred, chunked, deduplicated, prefetched, and pinned, and how new pods warm their working set before production traffic.
• Profile and reduce model load latency across the full path from disk to GPU memory, including the kernel, file system, NVMe-oF or RDMA transport, GPUDirect Storage, and inference-engine loader.
• Select, benchmark, and operate parallel and distributed file systems, object storage, NVMe drives and enclosures, and storage-management software; validate choices on Runware hardware before deployment.
• Set and meet reliability and durability targets; own capacity planning, monitoring, alerting, drive health and failure prediction, rebuild behavior, backup and recovery, and storage-layer on-call.
• Create a documented, automated pod bring-up playbook so storage is provisioned, validated, and warmed in hours rather than weeks.
• Partner with inference engine, platform, and hardware engineering teams to ensure storage is operable at scale and apply production learnings to future servers and pods.
• Use LLM-based agents to automate benchmarks, runbooks, triage, and tooling.
• Own data residency and operational storage security, including encryption at rest, access control, retention, and in-country placement for sovereign deployments.
• Travel to sites in the US and Europe as needed.

Benefits

• Meaningful stock options.
• Flexible scheduling: remote-first work, core collaboration hours, and flexibility to set your schedule outside them.
• Generous paid time off, including vacation, sick days, and public holidays.
• Paid maternity, paternity, and caregiver leave.
• Twice-yearly company retreats in inspiring locations.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed