AI Research Engineer

Dscout•San Francisco, CA

2h•$250,000 - $400,000•Remote

About The Position

Dscout’s AI Training & Evals unit uses consented human behavior and judgment to develop training data, evaluations, and benchmarks that help AI teams understand whether AI works for real people. This research-first role focuses on questions where a suitable rubric, benchmark, or method does not yet exist. The engineer identifies research gaps, proposes directions, and leads projects from hypotheses and experiments through benchmark construction and publication, while building methods that can operate within live customer programs. The role will develop science for evaluating rubrics, human raters, and AI judges; create a long-horizon benchmark based on real participants completing tasks in their own contexts; and study how human preference and judgment data can inform model training and evaluation. The benchmark is intended for public release, with the role owning its paper and release. The position also offers latitude to pursue related research, collaborate with external labs and academics, and translate findings into field-ready methods for delivery teams and forward deployed engineers. The position is based in the Bay Area. Dscout describes its workforce as remote-first and supports work from almost anywhere, with some location restrictions. Compensation is $250,000–$400,000 OTE plus equity. The package includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 company holidays, 12 weeks of paid parental leave, a 401(k) match, an education stipend, and a remote work stipend.

Requirements

Published research, or strong preprints and workshop papers, in areas such as evaluations, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
A PhD, a PhD in progress, or an MS plus at least two strong research internships at labs or top industry groups; creative machine-learning research experience without an advanced degree is also considered desirable.
Strong applied statistics, including inter-rater reliability, confidence intervals, and statistical power, and the ability to recognize misleading metrics.
Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
Strong Python skills and the ability to build research prototypes and evaluation pipelines that other people can run.
Use of AI coding tools as a real part of the workflow.
Experience running experiments on messy human data, not only curated datasets.
Clear technical writing and the ability to explain work to both lab researchers and product managers.
Based in the Bay Area.

Nice To Haves

Long-horizon or agentic evaluation experience.
Psychometrics, item response theory, or human-subjects research experience.
Contributions to open-source benchmarks or datasets.

Responsibilities

Design rubrics and test their validity, reliability, and sensitivity.
Measure human-rater reliability and distinguish noise from meaningful disagreement.
Calibrate AI judges against human judgment and study when they diverge.
Develop quality signals that provide evidence for data shipped to customers.
Design Dscout’s first long-horizon, real-user benchmark, including its protocol, tasks, scoring, and release.
Coordinate with the delivery team on benchmark fieldwork.
Maintain benchmark integrity over time by addressing drift, contamination, difficulty, and meaningful change.
Write the benchmark paper and own its public release.
Study when real-world human feedback improves models and when it introduces noise.
Investigate differences between preference and judgment data from representative people and expert data.
Translate research findings into methods forward deployed engineers and delivery teams can use in the field.
Propose and pursue new research programs, potentially including user-simulation calibration, human-AI collaboration, and behavioral evaluation of agents.
Co-author studies with external labs and academics that require real-user data.
Help run the paper-reading group.
Build research prototypes and evaluation pipelines that others can run.
Explain research clearly to lab researchers and product managers.