Site Reliability Engineer, Production

Thinking Machines Lab•San Francisco, CA

17h

About The Position

Thinking Machines is hiring a Site Reliability Engineer to drive the end-to-end reliability of Tinker, its fine-tuning API for researchers and developers. The role works alongside platform engineers and research teams to make the system more robust and resilient as Tinker adds customers, features, and use cases. Tinker manages infrastructure while giving users flexibility to fine-tune open-weight models with their own data and algorithms.

Requirements

Bachelor's degree or equivalent experience in computer science, engineering, or a similar field.
Experience in distributed systems, cloud infrastructure, or site reliability engineering.
Proficiency writing software to solve reliability problems, including building tooling and automation.
Experience with production incident response, postmortems, and systematic reliability improvement.
Strong communication skills and a track record of coordinating across engineering and research teams.

Nice To Haves

Deep experience operating production cloud services at scale, such as public cloud platforms or internal cloud services.
Background in distributed training frameworks and understanding how infrastructure failures surface in training behavior.
Track record building checkpoint and recovery systems for long-running distributed jobs.
Expertise deploying, operating, debugging, and tuning Kubernetes clusters at scale that handle heterogeneous GPU workloads.

Responsibilities

Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
Develop Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency with development velocity.
Design and implement monitoring and observability across the full training path.
Lead incident response for Tinker platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements to prevent recurrence.
Harden multi-tenant isolation and resource scheduling so LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation.
Collaborate with security teams to address production vulnerabilities.