Research, Post-Training Evals

Thinking Machines Lab•San Francisco, CA

15h

About The Position

Thinking Machines is hiring a researcher to develop reliable model evaluations that produce useful research signals. The role spans evaluation creation, usability, auditing, and efficiency, with close collaboration across post-training and the broader research organization. Depending on the candidate’s interests and experience, the work may focus on one area or span several.

Requirements

Bachelor’s degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline, with strong theoretical and empirical grounding.
Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.
Strong written and verbal communication skills and the ability to collaborate effectively across research and engineering teams.

Nice To Haves

Experience with LLMs, post-training, reinforcement learning, or agentic systems.
Experience with evaluation auditing, human evaluations, LLM judges, or open-ended task evaluation.
Experience with agentic evaluation, harnesses, long-horizon tasks, or RL environments.
Experience evaluating preferences, personalization, biases, values, or other nuanced model behaviors.
A track record of developing new evaluation methodologies or research signals that meaningfully influenced model development.
Proficiency in Python and familiarity with at least one deep learning framework, such as PyTorch, TensorFlow, or JAX.
Comfort debugging distributed training and writing code that scales.
Strong research judgment, including clean ablations, honest baselines, and clear technical writing.
PhD in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding, or equivalent industry research experience.

Responsibilities

Create internal evaluations and research signals for capabilities and behaviors important to model research and post-training.
Develop usability evaluations to assess whether models are genuinely useful in real research and product workflows.
Partner with the data flywheel to turn evaluation insights into improved data and training signals.
Improve evaluation robustness, including grader reliability, ambiguous ground truth, evaluator disagreement, false positives and negatives, and gaps between measured and intended behavior.
Build benchmark-auditing methodologies that help researchers understand, trust, and appropriately use evaluation signals.
Develop specialized agentic evaluation environments and user simulators, support harness development, and study generalization across users, harnesses, and environments.
Develop evaluations for personalized preferences, biases, values, and other nuanced dimensions of model behavior in collaboration with post-training crafting.