Research Engineer, Evals

Variance • San Francisco, CA

11h • Onsite

About The Position

Variance is building AI agents for high-stakes risk investigations, fraud, and identity reviews. The Research Engineer will define how the company measures and improves model quality by building benchmarks, datasets, tooling, and evaluation loops. The role sits at the intersection of research, product, and engineering, creating rigorous, domain-specific evaluations that reflect real customer workflows, reveal meaningful failure modes, and guide model and agent improvements.

The work includes developing evaluations for fraud, identity, and risk workflows; measuring model performance on real customer tasks; analyzing errors; and turning findings into stronger evaluations, datasets, and training loops. The engineer will also develop reusable evaluation tools and collaborate across teams to establish a culture of scientific experimentation, clear measurement, and continuous iteration.

Variance describes its culture as emphasizing ownership, urgency, and craft, with spirited debate and a fully in-person team in San Francisco.

Requirements

• Strong programming skills and comfort working in research-heavy codebases.
• Ability to design clean experiments and draw reliable conclusions from noisy results.
• Strong engineering judgment and a bias toward building.
• Care about craftsmanship, model quality, measurement, and experimental rigor.
• Interest in working on core model and agent behavior and defining quality in messy, high-stakes environments.
• Willingness to review outputs, grade edge cases, curate datasets, and refine evaluation tasks.
• Care about building systems that protect people from fraud, scams, and abuse.

Nice To Haves

• Experience training, evaluating, or improving modern ML systems.
• Experience building benchmarks, datasets, evaluation pipelines, or quality systems.
• Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas.
• Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains.

Responsibilities

• Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows.
• Design and run offline and online evaluations that measure model performance on real customer tasks.
• Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling.
• Study where models and agents fail, and use those findings to improve evaluations, datasets, and training loops.
• Build reusable evaluation tools and quality components for use across product surfaces and workflows.
• Partner with research, engineering, product, and design teams to improve system quality through rigorous experimentation.
• Help create a culture of scientific experimentation, clear measurement, and continuous iteration.
• Advance AI evaluation methods for regulated, adversarial, and high-consequence environments.

Benefits

• Competitive salary and meaningful equity.
• Platinum-level medical, dental, and vision insurance.
• Unlimited PTO, sick leave, and parental leave.
• Up to $100 per month in reimbursement for personal health and wellness expenses.
• 401(k) plan.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed