Evaluation Research Manager

Aaru•New York, NY

23h•Onsite

About The Position

Evaluation Research determines whether Aaru's populations, predictions, and end-to-end simulations correspond closely enough to the real world to support consequential decisions. The team defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities. The function builds both rails and carts. Rails are reusable evaluation infrastructure: datasets, harnesses, libraries, experiment standards, leaderboards, reporting systems, and ways to translate technical evidence into decisions. Carts are the specific evaluations that run on those rails: a historical backtest, a prospective forecast study, a population-coherence test, a reproduction of an observed behavioral pattern, or an end-to-end comparison with a resolved outcome. Evaluation Research is not conventional QA and it is not an internal approval service. It is an independent research function that works closely with the teams building Aaru's systems while preserving the ability to reach and communicate inconvenient conclusions.

Requirements

Led evaluation, measurement, or empirical research in machine learning, behavioral science, computational social science, statistics, economics, psychometrics, or a comparably rigorous environment.
Built evaluations that changed a research direction, model capability, product decision, or scientific conclusion.
Can define a difficult construct precisely enough to measure it without reducing away the underlying question.
Comfortable with experimental design, observational data, sampling, statistical power, uncertainty, causal threats, leakage, and condition shift.
Can write code, analyze large datasets, design studies, inspect individual failures, and review the technical work of researchers and engineers.
Can work closely with builders while reaching independent conclusions about the quality of their systems.
Care more about an accurate result than a favorable one and are willing to revise your own evaluation when evidence shows it is inadequate.
Can prioritize a research portfolio and choose which uncertainty is most important to resolve next.
Managed or technically led strong researchers, give clear feedback, and can develop independent judgment rather than creating dependence on your review.
Can explain technical evidence clearly to researchers, engineers, product teams, customers, company leadership, and non-specialists.
Want to work in person in New York with a team that moves quickly and takes truth-seeking seriously.

Nice To Haves

Work in ML evaluation, model behavior, forecasting, econometrics, psychometrics, causal inference, experimental economics, survey methodology, or measurement theory.
Experience evaluating LLM agents, multi-agent systems, synthetic populations, recommender systems, probabilistic models, simulations, or decision-support tools.
Experience with longitudinal records, transaction data, product analytics, field experiments, prospective studies, backtesting, or validation against operational outcomes.
Experience building evaluation platforms, regression suites, experiment-tracking systems, shared research datasets, model scorecards, or scientific reporting tools.
A record of finding an important failure that standard metrics missed and developing a better measurement method.
Experience communicating scientific results in customer-facing, public, policy, legal, or regulatory settings.
Experience hiring and leading a small, high-talent research team through ambiguous work with short feedback cycles.
A PhD, provided you have equivalent evidence of rigorous empirical work and research leadership.
Managed managers or a large organization; this role is about leading a focused team and remaining directly involved in the research.

Responsibilities

Build, lead, and develop a high-performing team of Evaluation Researchers and research engineers.
Turn broad questions about realism, accuracy, calibration, usefulness, and decision quality into measurable constructs, decisive experiments, and explicit decision criteria.
Set a focused evaluation agenda across population construction, predictive systems, individual agent behavior, group dynamics, and end-to-end simulations.
Decide which evaluation infrastructure should become a reusable organizational rail and which questions require a purpose-built study.
Establish standards for baselines, temporal holdouts, prospective testing, contamination control, statistical power, uncertainty, subgroup analysis, and reproducibility.
Build tests of population quality that assess individual coherence, joint and conditional distributions, representation of rare but plausible profiles, and whether a profile induces behavior consistent with the person it represents.
Evaluate forecasts and other predictive outputs using calibration, proper scoring rules, ranking quality, selective prediction, temporal validity, subgroup performance, and the real cost of different errors.
Compare simulations with transactions, product usage, behavioral traces, operational outcomes, market movements, resolved events, and longitudinal decisions.
Design end-to-end studies that determine whether improvements to a component actually improve the decision-relevant output customers receive.
Find failures hidden by aggregate metrics, especially those concentrated in important subgroups, rare cases, changing environments, or ambiguous ground truth.
Create diagnostic evaluations that help researchers localize why a system failed and distinguish a real general improvement from benchmark-specific optimization.
Partner with Simulation Engineering to make evaluations repeatable, versioned, scalable, and integrated into development and release workflows without compromising protected holdouts.
Convert production incidents, customer surprises, and deployment failures into durable test cases and better measurement methods.
Review evidence used in product, customer, or public claims and ensure that conclusions are reproducible, appropriately scoped, and honest about uncertainty and limits.
Communicate negative, null, and inconclusive results with the same precision and urgency as positive findings.
Recruit exceptional researchers, set clear expectations, provide direct feedback, develop independent research judgment, and address performance problems early.