AI Research Engineer

Dscout•San Francisco, CA

5d•Remote

About The Position

Dscout AI Training & Evals builds consented human-behavior data, evaluations, and benchmarks to help AI teams determine whether AI works for real people in real-world contexts. The AI Research Engineer is a research-first role focused on questions for which the appropriate rubric, benchmark, or method does not yet exist. The engineer identifies research gaps, proposes directions, and leads projects from initial hypotheses through experiments, benchmark construction, and publication. The work also needs to operate within live customer programs. The role will develop and validate rubrics and quality-assurance methods; design Dscout’s first long-horizon, real-user benchmark; and study how human judgment can serve as training and evaluation signal. The engineer will work with delivery teams on fieldwork, create methods that colleagues can use independently, and own the benchmark paper and public release. There is latitude to pursue additional research directions and collaborate with external labs and academics. Dscout describes the team as small and senior, with direct access to the Head of AI Research and the GM, and says it works AI-natively using agents and evaluations in its development workflow. A successful first year would include a validated QA and rubric framework in live customer programs, a released benchmark v0 with a paper submitted to a top venue, external use or citation of the work, and methods that delivery colleagues can run without the engineer present. The position is based in the Bay Area. Dscout supports a remote-first workforce, but cannot currently hire in Montana, Hawaii, Alaska, or Washington, DC.

Requirements

Published research, or strong preprints and workshop papers, in areas such as evaluations, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
A PhD, a PhD in progress, or an MS plus at least two strong research internships at labs or top industry groups; creative machine-learning research work without an advanced degree is also considered desirable.
Strong applied statistics skills, including inter-rater reliability, confidence intervals, and statistical power, and the ability to recognize misleading metrics.
Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
Strong Python skills and the ability to build research prototypes and evaluation pipelines that others can run.
Use of AI coding tools as a substantive part of the work process.
Experience running experiments on messy human data, not only curated datasets.
Clear technical writing and the ability to communicate work to both research and product audiences.
Based in the Bay Area.

Nice To Haves

Long-horizon or agentic evaluation work.
Psychometrics, item response theory, or human-subjects research.
Contributions to open-source benchmarks or datasets.

Responsibilities

Identify important research gaps and propose research directions where rubrics, benchmarks, or methods do not yet exist.
Lead research projects from hypothesis through experiments, benchmark construction, and publication.
Design rubrics and assess their validity, reliability, and sensitivity.
Measure human-rater reliability and distinguish noise from meaningful disagreement.
Calibrate AI judges against human judgment and study when they diverge.
Build evidence-based quality signals for data delivered by Dscout.
Design Dscout’s first long-horizon benchmark using real users in real contexts, including multi-week tasks across tools.
Define the benchmark protocol, task design, scoring, and release; coordinate fieldwork with the delivery team.
Maintain benchmark quality over time by addressing drift, contamination, difficulty, and meaningful change.
Write the benchmark paper and own its public release.
Study when real-world human feedback improves models and when it introduces noise.
Compare preference and judgment data from representative people with expert data.
Develop field-ready methods for forward deployed engineers and the delivery team.
Propose new research programs, potentially including user-simulation calibration, human-AI collaboration, and behavioral evaluation of agents.
Co-author studies with external labs and academics that require real-user data.
Help run the paper-reading group.
Explain research clearly to lab researchers and product managers.