(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure

Together AI•San Francisco, CA

13h

About The Position

Together AI is seeking a reliability-minded software engineer to own the production reliability of the Kubernetes-native control plane that provisions and operates its GPU inference fleet. The platform provides a manifest-driven API and controllers through which the inference team declares infrastructure needs and has them reconciled. The engineer will keep that system reliable in practice while preserving decoupled platform abstractions and improving fleet efficiency without introducing instability. The role owns SLOs and error budgets for inference serving and provisioning, leads incident response, and translates postmortem findings into changes to controllers and automation. It involves troubleshooting InfiniBand/RDMA and model-serving failures, extending automated detection and remediation, operating the host lifecycle pipeline, and partnering with inference and ML platform teams to improve platform abstractions based on real-world failure modes. The engineer will also develop tools and architectural improvements that increase inference performance, latency, throughput, and efficiency, and help align reliability priorities across infrastructure, ML platform, and product teams.

Requirements

Experience operating a provisioning or host-lifecycle pipeline in production and troubleshooting failed or stuck state transitions at fleet scale.
Deep experience owning production reliability for large-scale distributed systems.
Hands-on experience troubleshooting InfiniBand/RDMA networking, ideally in multi-node training or disaggregated inference; able to work with IB port counters and subnet manager logs.
Familiarity with multi-tenant model serving, including LoRA/adapter-based serving, and differentiated SLOs across model pools of different priority.
Track record of eliminating operational toil through automation and tooling built directly.
Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
Comfort with event-driven architectures, backpressure and circuit-breaking design, and integrating metrics, logs, and traces into systems.
Software-engineering mindset: writes and ships production code and treats GPU infrastructure as software to engineer.

Nice To Haves

GPU/ML-serving infrastructure experience.
InfiniBand/RDMA troubleshooting experience in a multi-node training or disaggregated-inference context.

Responsibilities

Operate the provisioning state machine in production, owning the host lifecycle pipeline from discovery and inference bring-up through GPU driver/CUDA validation, health checks, decommissioning, and RMA.
Triage stuck or failed state transitions at fleet scale and feed observed failure patterns into improvements to state-machine design.
Diagnose failures missed by the node-scanning and remediation platform, including InfiniBand issues, GPU Xid variants, and hero-model or LoRA-specific faults; deliver fixes and add detection/remediation plugins.
Own SLOs and error budgets for provisioning and fleet health, lead incident response, and drive systemic fixes when idempotency, retry, rollback, or drift assumptions fail.
Partner with inference and ML platform teams to translate real cluster topology, interconnect, and scheduling failure modes into platform requirements.
Work with machine learning researchers, engineers, and product managers to bring new technologies into production and enable advanced research through engineering.
Introduce techniques, tools, and architecture to improve model-inference performance, latency, throughput, and efficiency.
Build visibility tools to identify bottlenecks and instability, then design and implement solutions to priority issues.
Lead reliability efforts across infrastructure, ML platform, and product teams; manage stakeholders during incidents and roadmap trade-offs, and represent reliability in planning.