(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure

Together AI•San Francisco, CA

1d•$250,000 - $320,000

About The Position

Together AI is seeking a reliability-minded software engineer to own production reliability for the Kubernetes-native control plane that provisions and runs its GPU inference fleet. The platform provides a manifest-driven API and controllers through which inference teams request clusters, model deployments, and capacity changes. This role ensures those requests are reliably reconciled in production, including diagnosing fleet-scale lifecycle failures and specialized inference issues, and extending automation so newly discovered failure modes are addressed permanently. The engineer will own SLOs and error budgets for inference serving and provisioning, lead incident response, and translate postmortem findings into changes to controllers and automation. The role also addresses the reliability of capacity efficiency, including the risks of defragmentation and bin-packing, and partners with inference and ML platform teams to make production failure modes first-class requirements in platform abstractions. The position works with researchers, engineers, and product managers to bring new technologies into production and improve inference performance, latency, throughput, and efficiency. It also builds visibility and remediation tools, improves the on-call experience and automated responses, and leads cross-functional reliability priorities and trade-offs. Together AI describes itself as an AI Native Cloud providing inference, fine-tuning, reinforcement learning, pre-training, and a marketplace of open models. The company says it serves more than 400 trillion tokens per month and is trusted by companies including Cursor, Decagon, ElevenLabs, Salesforce, and Zoom.

Requirements

Experience operating a provisioning or host-lifecycle pipeline in production, including troubleshooting fleet-scale state transitions such as discovery, bring-up, GPU driver/CUDA validation, health checks, and decommission/RMA, and translating failures into state-machine improvements.
Deep experience owning production reliability for large-scale distributed systems.
Hands-on experience troubleshooting InfiniBand/RDMA networking issues; ability to read InfiniBand port counters and subnet manager logs.
Familiarity with operational realities of multi-tenant model serving, including LoRA/adapter-based serving and differentiated SLOs across model pools of different priority.
Demonstrated track record of eliminating operational toil through self-built automation and tooling.
Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health and remediation systems.
Comfort with event-driven architectures, backpressure and circuit-breaking design, and observability through metrics, logs, and traces.
Software-engineering mindset: writes and ships production code and treats GPU infrastructure as software to be engineered.

Nice To Haves

GPU/ML-serving infrastructure experience is a strong plus.
Experience troubleshooting InfiniBand/RDMA in a multi-node training or disaggregated-inference context.

Responsibilities

Operate the provisioning state machine in production and own the health of the host lifecycle pipeline, including discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommission/RMA.
Triage fleet-scale stuck or failed state transitions and feed observed failure patterns into state-machine design improvements.
Diagnose failures that automated node-scanning and remediation systems miss, including InfiniBand issues in disaggregated prefill/decode serving, GPU Xid variants, and hero-model- or LoRA-specific faults; implement fixes and detection/remediation plugins.
Own end-to-end reliability of provisioning and fleet health, including SLOs and error budgets; lead incident response and drive systemic fixes when idempotency, retry, rollback, or drift assumptions fail.
Partner with inference and ML platform teams to communicate production failure modes and make real cluster topologies, interconnects, and scheduling constraints first-class platform requirements.
Work with machine learning researchers, engineers, and product managers to bring new technologies into production and enable research through engineering.
Introduce techniques, tools, and architecture to improve model-inference performance, latency, throughput, and efficiency.
Build tools to expose bottlenecks and instability, then design and implement solutions to high-priority issues.
Lead reliability work across infrastructure, ML platform, and product teams; align stakeholders during incidents and roadmap trade-offs, and represent reliability in planning.
Improve the on-call experience, alert clarity, and automated-remediation decisions through ongoing product-minded iteration.