Research Engineer / Scientist, Data & Evals

Harvey

6d • $230,000 - $340,000

About The Position

As a Member of Technical Staff, Data & Evals in Harvey Research, you will build the data and measurement systems that support post-training, benchmarking, and agent capability work. Harvey has post-trained its first model built for legal work and is growing its team to scale post-training efforts. The role owns synthetic data pipelines, evaluation design, tasks and grading for training environments, and systems that turn domain-expert feedback into training signals and rewards. Because correctness in these tasks often cannot be verified automatically, defining and measuring quality is a core research challenge. You will work on a small, high-bar team of hands-on engineers and researchers, partnering with researchers and legal experts to turn model failures into measurable improvements.

Requirements

• Experience building data pipelines, ML evaluation systems, or experimentation tooling.
• Strong Python skills and ownership of production-quality code.
• A rigorous approach to measurement, including statistical reasoning, experimental design, and understanding when metrics can mislead.
• Ability to translate ambiguous goals or expert judgment into structured datasets, rubrics, and measurable criteria.

Nice To Haves

• Experience with synthetic data, LLM evaluations, automated raters, or preference data.
• Strong data and measurement engineering experience close to ML model development; prior LLM work is not required.

Responsibilities

• Build synthetic data pipelines that generate, filter, and curate training and evaluation samples for complex legal reasoning and agentic workflows.
• Design evaluations and verifiers that reflect real legal work, calibrate them against expert judgment, and investigate their failures.
• Build and maintain capability benchmarks, including the Legal Agent Benchmark (LAB), and product-focused evaluations that track model performance on real customer workflows.
• Work with legal domain experts to define quality metrics, rubrics, and other grading mechanisms for training and evaluation.
• Own training and evaluation data quality, including provenance, coverage, and safeguards against contamination.
• Connect product usage and customer feedback to new evaluations and training data so models improve in areas customers need most.
• Partner with researchers to investigate model failures, distinguish real gains from misleading metrics, and use findings to guide training and release decisions.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed