Evaluation Researcher

Aaru•New York, NY

8d•Onsite

About The Position

Evaluation Research determines whether Aaru's populations, predictions, and end-to-end simulations correspond closely enough to the real world to support consequential decisions. The team defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities. The function builds both rails and carts. Rails are reusable evaluation infrastructure: datasets, harnesses, libraries, experiment standards, leaderboards, reporting systems, and ways to translate technical evidence into decisions. Carts are the specific evaluations that run on those rails: a historical backtest, a prospective forecast study, a population-coherence test, a reproduction of an observed behavioral pattern, or an end-to-end comparison with a resolved outcome. Evaluation Research is not conventional QA and it is not benchmark administration. It is an independent research function. The work requires understanding the systems deeply, collaborating closely with their builders, and remaining willing to conclude that an attractive method did not improve what matters.

Requirements

A record of rigorous work in ML evaluation, statistics, behavioral science, computational social science, economics, psychometrics, experimental design, or a related field.
You have built an evaluation that changed a research direction, model capability, product decision, or scientific conclusion.
You can define a difficult construct precisely enough to measure it without losing the underlying question.
You are comfortable with observational data, sampling, statistical power, uncertainty, causal threats, selection, leakage, and condition shift.
You can write code, analyze large datasets, build evaluation systems, and inspect individual examples rather than relying only on aggregate dashboards.
You naturally ask what a benchmark actually measures, what behavior it incentivizes, and where it can be gamed or contaminated.
You can work closely with a system's builders while maintaining independent judgment about its quality.
You care more about an accurate conclusion than a favorable one and are willing to revise your own evaluation when it proves inadequate.
You communicate technical evidence clearly to researchers, engineers, product teams, customers, and non-specialists.
You want to work in person in New York with a team that moves quickly and treats inconvenient evidence as valuable.

Nice To Haves

Work in forecasting evaluation, econometrics, psychometrics, causal inference, survey methodology, experimental economics, measurement theory, or model behavior.
Experience evaluating LLM agents, multi-agent systems, synthetic populations, recommender systems, probabilistic models, simulations, or decision-support tools.
Experience with longitudinal records, transaction data, product analytics, field experiments, prospective studies, or operational validation.
Experience building evaluation platforms, regression suites, experiment-tracking systems, shared datasets, model scorecards, or scientific reporting tools.
Experience with automated graders, human evaluation, rubric design, inter-rater reliability, benchmark contamination, or adversarial evaluation.
A record of finding an important failure that standard metrics missed and developing a better way to measure it.
Experience communicating scientific results in customer-facing, public, policy, legal, or regulatory settings.
Prior experience in population simulation or a job title containing the word “evaluation.”
A PhD, provided you can demonstrate equivalent research depth and empirical rigor.
Expertise in every statistical or machine-learning method listed above.

Responsibilities

Own an important evaluation or measurement area across population construction, predictive systems, individual agent behavior, group dynamics, or end-to-end simulations.
Turn broad questions about realism, accuracy, calibration, usefulness, and decision quality into measurable constructs and explicit decision criteria.
Design studies using historical backtests, temporal holdouts, prospective outcomes, observational records, controlled experiments, expert judgment, or mixed methods as the problem requires.
Build tests of individual-profile quality, including internal coherence, contradictions, impossible combinations, unsupported specificity, stability, and whether a profile induces behavior consistent with the represented person.
Build tests of population quality, including marginal and joint distributions, conditional relationships, coverage of rare but plausible profiles, subgroup fidelity, and sensitivity to sampling choices.
Evaluate predictions using calibration, proper scoring rules, ranking quality, selective prediction, temporal validity, subgroup performance, and the decision cost of different errors.
Compare simulations with transactions, product usage, behavioral traces, operational outcomes, market movements, resolved events, surveys, and longitudinal decisions.
Design longitudinal and interaction-based evaluations that test how agents change over time, respond to new information, and influence one another.
Build end-to-end studies that determine whether a component improvement actually changes the quality of the conclusion a customer receives.
Establish strong baselines and compare agent-based simulation with direct forecasting, conventional statistical models, simpler segment-level methods, and human or market benchmarks where appropriate.
Find failures hidden by aggregate metrics, especially failures concentrated in important subgroups, rare cases, changing environments, or ambiguous labels.
Build diagnostic evaluations that localize why a system failed and whether a proposed fix generalizes beyond the development set.
Develop reusable evaluation datasets, harnesses, libraries, graders, experiment schemas, leaderboards, and reporting tools where shared infrastructure will accelerate future research.
Work with Simulation Engineering to version and automate evaluations while keeping protected holdouts and final claims insulated from development leakage.
Convert customer surprises, production incidents, and resolved real-world outcomes into durable test cases.
Write clear technical reports that distinguish exploratory evidence from claim-supporting evidence and communicate uncertainty, limitations, and alternative interpretations.
Report negative, null, and inconclusive findings with the same care as positive results.