(Senior/Junior AI Inference Reliability Engineer, Compute Infra

Together AI•San Francisco, CA

21h

About The Position

Together AI is seeking a software engineer to own production reliability for the Kubernetes-native control plane that provisions and runs its GPU inference fleet. The platform uses a manifest-driven API and controllers to reconcile requests for clusters, model deployments, and capacity changes. This role keeps that system reliable in production while maintaining the separation between platform and inference systems and ensuring fleet-efficiency measures do not introduce instability. The engineer will operate and improve provisioning and host-lifecycle systems, investigate GPU-serving and InfiniBand/RDMA failures, and extend automated detection and remediation so recurring failure modes are addressed through lasting code and system changes. The role owns SLOs and error budgets for provisioning and fleet health, leads incident response, and turns postmortem findings into controller and automation improvements. It also addresses reliability risks arising from capacity-efficiency mechanisms such as defragmentation and bin packing. Working with inference and ML platform teams, researchers, engineers, and product managers, the engineer will bring new technologies into production, improve inference performance and efficiency, and build tools that reveal bottlenecks and instability. The position requires a software-engineering and product mindset, including iterating on alerts, on-call experience, and automated remediation. The US base salary range is $160,000–$280,000, plus equity and benefits; compensation depends on location, level, experience, skills, and job-related knowledge.

Requirements

Experience operating a production provisioning or host-lifecycle pipeline and troubleshooting fleet-scale state transitions such as discovery, bring-up, GPU driver/CUDA validation, health checks, and decommission/RMA.
Deep experience owning production reliability for large-scale distributed systems.
Hands-on experience troubleshooting InfiniBand/RDMA networking issues; familiarity with reading InfiniBand port counters and subnet-manager logs.
Familiarity with multi-tenant model serving, including LoRA/adapter-based serving, and differentiated SLOs for model pools of varying priority.
Demonstrated ability to eliminate operational toil through self-built automation and tooling.
Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
Comfort with event-driven architectures, backpressure and circuit-breaking design, and building observability (metrics, logs, and traces) into systems.
Software-engineering mindset and experience writing and shipping production code; treats GPU infrastructure as software to engineer.

Nice To Haves

GPU/ML-serving infrastructure experience.
InfiniBand/RDMA troubleshooting in a multi-node training or disaggregated-inference context.

Responsibilities

Operate the provisioning state machine in production and own the health of the host-lifecycle pipeline, including discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommissioning, and RMA.
Triage stuck or failed state transitions at fleet scale and incorporate real failure patterns into the design of states and transitions.
Diagnose production failures missed by existing node-scanning and remediation systems, including InfiniBand issues, GPU Xid variants, and hero-model- or LoRA-specific faults; drive fixes into production and add detection and remediation plugins.
Own end-to-end reliability of provisioning and fleet health, including SLOs and error budgets; lead incident response and drive systemic fixes when idempotency, retries, rollbacks, or drift assumptions fail.
Partner with inference and ML platform teams to translate production failure modes, cluster topologies, interconnects, and scheduling constraints into platform requirements.
Work with machine-learning researchers, engineers, and product managers to bring new technologies into production and enable research through engineering.
Introduce techniques, tools, and architecture to improve inference performance, latency, throughput, and efficiency.
Build visibility tools to identify bottlenecks and instability, then design and implement solutions to high-priority issues.
Lead reliability work across infrastructure, ML platform, and product teams; align stakeholders during incidents and roadmap trade-offs, and represent reliability in planning.
Improve the reliability of capacity-efficiency mechanisms, including defragmentation and bin-packing passes.