(Senior/Junior AI Inference Reliability Engineer, Compute Infra

Together AI•San Francisco, CA

1d•$160,000 - $280,000

About The Position

Together AI is seeking a software engineer with a reliability mindset to own the production reliability of the Kubernetes-native control plane that provisions and runs its GPU inference fleet. The platform provides manifest-driven APIs and controllers through which teams declare cluster, model-deployment, and capacity needs; this role helps ensure those needs are reliably reconciled in production. The engineer will diagnose issues involving InfiniBand and RDMA, GPU and model-serving faults, and failures in provisioning and host-lifecycle state transitions. They will extend automated detection and self-healing so that recurring failure modes become permanent fixes. The role owns SLOs and error budgets for inference serving and provisioning, leads incident response, and turns postmortem findings into changes to controllers and automation. It also addresses capacity efficiency while ensuring defragmentation and bin-packing do not introduce instability. The engineer partners with inference and ML platform teams, researchers, engineers, and product managers to improve the performance, latency, throughput, and efficiency of the inference stack. The role emphasizes shipping production software and treating the on-call experience, alerts, and automated remediation as products to improve over time.

Requirements

Experience operating a provisioning or host-lifecycle pipeline in production, including troubleshooting stuck or failed state transitions at fleet scale and improving the state machine based on observed failures.
Deep experience owning production reliability for large-scale distributed systems.
Hands-on experience troubleshooting InfiniBand/RDMA networking issues; ideally in multi-node training or disaggregated-inference settings, including reading IB port counters and subnet manager logs.
Familiarity with multi-tenant model serving, including LoRA/adapter-based serving, and differentiated SLOs across model pools of different priority.
A demonstrated track record of eliminating operational toil with self-built automation and tooling.
Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
Comfort with event-driven architectures, backpressure and circuit-breaking design, and building observability (metrics, logs, and traces) into systems.
A software-engineering mindset and experience writing and shipping production code.

Nice To Haves

GPU/ML-serving infrastructure experience.
InfiniBand/RDMA troubleshooting experience in a multi-node training or disaggregated-inference context.

Responsibilities

Operate the provisioning state machine in production and own the health of the host-lifecycle pipeline, including discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommissioning/RMA.
Triage stuck or failed state transitions at fleet scale and feed observed failure patterns into improvements to state and transition design.
Diagnose failures that automated node-scanning and remediation systems miss, including InfiniBand issues, GPU Xid variants, and hero-model- or LoRA-specific faults; implement fixes and detection/remediation plugins.
Own end-to-end reliability of provisioning and fleet health, including SLOs and error budgets; lead incident response and drive systemic fixes when idempotency, retry, rollback, or drift assumptions fail.
Partner with the inference and ML platform teams to translate real-world cluster topology, interconnect, and scheduling failure modes into platform requirements.
Work with machine learning researchers, engineers, and product managers to bring new technologies into production and enable research through engineering.
Introduce techniques, tools, and architecture that improve inference performance, latency, throughput, and efficiency.
Build tools to identify bottlenecks and instability, then design and implement solutions to the highest-priority issues.
Lead reliability work across infrastructure, ML platform, and product teams; coordinate stakeholders during incidents and roadmap trade-offs, and represent reliability in planning.
Improve the on-call experience, alert clarity, and automated-remediation decisions through continued iteration.