Software Engineer, Production Inference (Distributed Inference)

Thinking Machines Lab•San Francisco, CA

3d•$350,000 - $500,000•Onsite

About The Position

We're hiring a Software Engineer to build and scale the distributed production inference systems that serve Inkling, Inkling-Small, and Tinker in production. You'll own the systems that turn trained models into fast, reliable, cost-efficient services — from request routing and batching to multi-node serving and GPU utilization at scale. This is a systems-heavy, production-first role. You'll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems that meet real-world latency, throughput, and reliability requirements, and you'll be on the front line when production inference systems need to scale, recover, or improve.

Requirements

3+ years of experience building and operating distributed systems in production
Strong systems programming skills in Python, C++, Rust, or similar languages
Experience with production infrastructure at scale: reliability, observability, and performance under real-world load
Solid understanding of networking, concurrency, and distributed systems fundamentals

Nice To Haves

Experience with LLM inference engines such as vLLM, SGLang, or TensorRT-LLM
Familiarity with GPU programming (CUDA) or low-level performance optimization
Experience with model parallelism, tensor/pipeline parallelism, or other distributed inference techniques
Track record of operating large-scale production systems with strict latency and uptime requirements
Experience with Kubernetes or similar orchestration systems for GPU workloads
Contributions to open-source ML systems or inference infrastructure projects

Responsibilities

Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
Participate in on-call rotation to support production inference systems