Member of Technical Staff, Evals

Magic•San Francisco, CA

10d•$200,000 - $550,000

About The Position

Evals builds the internal platform that teams across Magic use to evaluate the performance of first-party and third-party models. The team supports pre-training, post-training, data, inference, and product, and sits on the critical path of many of the company's most important decisions. As a Member of Technical Staff on Evals, you will build both the platform and the evaluations themselves. You'll develop infrastructure for large-scale evaluations, data ablations, and dataset quality analysis, while designing and validating the methodologies used to measure model performance. Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality. Evals are essential to the success of the company. By building trustworthy evaluation systems, you will help Magic make better research decisions, build better datasets, and ship better products.

Requirements

Strong software engineering fundamentals
Experience building production systems, internal platforms, or developer infrastructure
Exceptional attention to detail and a high bar for correctness
Experience working with machine learning systems, evaluation frameworks, data infrastructure, or research tooling
Ability to reason critically about benchmarks, metrics, and experimental methodology
Strong intuition for measurement quality and experimental design
Experience designing, implementing, or operating systems that run at scale
Strong debugging and investigative skills
Comfortable navigating ambiguity and determining whether a measurement is actually capturing the behavior it claims to measure
Skepticism toward results that cannot be reproduced, validated, or explained
Track record of owning technical projects end-to-end
Excitement about helping researchers and engineers make better decisions through trustworthy measurements

Responsibilities

Build and maintain the internal evals platform used across Magic
Design, implement, and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems
Develop infrastructure for running large-scale evaluations
Build systems to measure dataset quality and identify opportunities to improve training data
Improve evaluation correctness, reproducibility, and reliability
Audit and improve upon public benchmarks, evaluation methodologies, and open-source implementations
Partner with research, data, inference, and product teams to define metrics that accurately reflect model quality
Build tooling and frameworks that enable teams across Magic to make decisions based on trustworthy measurements