Software Engineer, Evaluation Platform / Infra

Thinking Machines Lab•San Francisco, CA

2d•Onsite

About The Position

Thinking Machines is building AI intended to extend human will and judgment, including frontier models, tools for customizing models, and interfaces for human-AI communication. Evaluation is central to the company’s AI research: it guides research direction, supports experimentation, and helps determine whether changes to data and training improve model capabilities and behaviors. The Software Engineer will design and build an end-to-end evaluation platform that enables researchers to author evaluations, run and reproduce them reliably at scale, and analyze results. The platform must support external benchmarks and internal evaluations, varied tasks and graders, and inspection ranging from aggregate metrics to individual model trajectories. The role spans Python frameworks, data pipelines, APIs, distributed backend systems, and user-facing applications. The engineer will collaborate with pre-training, post-training, and applied teams, and work with Research Tooling, the team behind Thinking Machines’ internal research platform.

Requirements

A bachelor’s degree or equivalent practical experience in computer science, engineering, machine learning, or a related field.
Two years of post-graduate work experience as a software engineer or ML engineer, excluding internships.
Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
Strong software engineering fundamentals and experience building reliable, maintainable systems.
Proficiency in at least one backend programming language; the company primarily uses Python and Rust.
Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
Comfort working across the stack and owning projects from initial problem discovery through deployment and operation.
Experience collaborating with cross-functional partners and subject-matter experts.

Nice To Haves

A track record of building frameworks, SDKs, or developer tools with thoughtful abstractions and a strong user experience.
Experience with distributed job execution, workflow orchestration, sandboxed environments, or large-scale data processing.
Experience building polished, intuitive interfaces for inspecting complex data, comparing experiments, or debugging model behavior.
Familiarity with large language or multimodal model evaluation, including model-based grading, human evaluation, or synthetic data.
Experience working closely with researchers to understand their workflows and turn rapidly evolving needs into durable systems.
Experience at a startup or on a small team, building technically complex products end to end.

Responsibilities

Design, build, and maintain the platform for authoring, running, tracking, and analyzing model evaluations used in daily work and model releases.
Work across evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications to deliver capabilities end to end.
Build flexible abstractions for evaluation tasks, environments, graders, datasets, and model outputs without constraining fast-moving research.
Make evaluation results reproducible and trustworthy through versioning, provenance, observability, failure recovery, and robust quality controls.
Partner directly with researchers to identify bottlenecks and turn bespoke workflows into self-serve systems that work across teams.
Collaborate with Research Tooling, the engineering team behind Thinking Machines’ internal research platform.
Collaborate with pre-training, post-training, and applied teams to improve model evaluation and translate results into research decisions.