Software Engineer - ML Infrastructure

Watney • San Francisco, CA

2d

About The Position

At Watney, ML Infrastructure software engineers build the high-performance foundations that allow our perception and intelligence models to scale. You will architect the high-performance computing foundation that powers our physical intelligence models. In this role, you’ll own the infrastructure required for large-scale multimodal training, which includes cluster orchestration, optimizing JAX-based pipelines that must ingest and stream video data, and transforming experimental architectures into reliable, highly distributed production training runs. This is a high-leverage systems role at the intersection of deep learning, advanced hardware acceleration, and scalable cluster infrastructure.

Requirements

• Experience building machine learning platforms and large-scale distributed training
• Deep professional experience with distributed training backbones (FSDP, DeepSpeed, Megatron, Ray Train) or large-scale inference serving layers (vLLM, Triton, Ray Serve).
• Fluency in Python alongside Rust or C/C++
• Strong mathematical background
• Practical knowledge of GPU kernel optimization or network topologies.
• Experience navigating structural edge-case hardware bottlenecks, specifically regarding video decoding, multimodal alignment, or high-throughput real-time playback.

Responsibilities

• Own Training & Inference Infrastructure: Design and maintain multi-tenant scheduling systems that automatically place training and inference jobs based on hardware topology, cost, and priority, while enforcing fair resource sharing and preemption policies.
• Scale Distributed Training: Partner with researchers to scale JAX and PyTorch-based training loops across heterogeneous GPU/TPU clusters with minimal friction, ensuring rock-solid checkpointing and metrics collection.
• Optimize Performance & Hardware Bounds: Profile and improve memory usage, device utilization, throughput, and distributed synchronization, specifically navigating edge hardware bottlenecks like on-chip video decoders and memory bandwidth.
• Enable Rapid Iteration: Build clean abstractions for launching, monitoring, debugging, and reproducing experiments so researchers can submit massive jobs without needing to manage underlying cluster state.
• Contribute to Core Training Code: Evolve our core JAX model code and training pipelines to natively support new architectures, multimodal video/telemetry data streams, and robust evaluation metrics.
• Manage Compute Resources: Ensure highly efficient allocation and utilization of massive cloud-based compute clusters while aggressively monitoring and controlling resource costs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed