Evaluations - Member of Technical Staff

Simile•San Francisco, CA

About The Position

Simile is seeking a Member of Technical Staff, Model Evaluations to build the measurement systems that determine the accuracy, trustworthiness, and usefulness of their AI simulations of human behavior. This role involves shaping what Simile measures, defining quality standards, and guiding decisions based on evaluation evidence. The position requires a blend of expertise in model evaluation, statistics, behavioral science, research methodology, product quality, and human judgment. The candidate will work with rich behavioral data to assess simulations of people, populations, markets, and groups, requiring reasoning about distributions, noisy ground truth, uncertainty, and qualitative outputs. The core question for this role is determining when a simulation of human behavior is reliable enough for real-world decisions.

Requirements

Evaluation Taste: Strong intuition for meaningful, robust, and decision-relevant evaluations; ability to explain what an eval measures, its limitations, potential for gaming, and its impact on model/product decisions.
LLM and Model Fluency: Understanding of modern LLM training, post-training, evaluation, and optimization; ability to read model outputs, understand modeling team needs, and assess improvements in relevant capabilities.
Statistical Judgment: Comfort reasoning about noisy data, uncertainty, sampling, distributions, calibration, confidence intervals, measurement validity, bias, variance, and the distinction between observed results and underlying population quantities.
Technical and Agentic Execution: Ability to rapidly build internal tools, scripts, dashboards, labeling workflows, analyses, or automated eval pipelines using Python, SQL, R, notebooks, LLM APIs, and agentic coding tools (e.g., Codex, Claude Code, Cursor); skill in moving quickly while validating outputs, catching errors, and planning for the long-term.
Hands-On Ownership: Ability to independently drive workstreams while performing the work; willingness to build initial versions, inspect data, debug workflows, write rubrics, revise metrics, and ensure evaluation systems are useful.

Nice To Haves

Modeling / Model-Quality Dashboards: Experience building model evaluation dashboards, regression suites, release gates, benchmark sets, model comparison workflows, or systems to guide ML teams.
LLM-as-Judge and Human Data: Experience designing rubrics, automated graders, pairwise comparisons, expert review workflows, labeling interfaces, grader calibration, or human/model hybrid evaluation systems.
Survey Methodology and Statistics: Experience with sampling, weighting, margin of error, power analysis, uncertainty quantification, Bayesian modeling, causal inference, psychometrics, polling, or measurement theory.
Behavioral Simulation: Experience evaluating behavioral predictions beyond self-reported survey responses, using data like transactions, purchase behavior, mobility, product interactions, or other passively collected behavioral signals.
Behavioral Economics / Experimentation: Experience designing RCTs, A/B tests, survey experiments, vignette studies, field experiments, behavioral games, or intervention studies.
Multi-Agent or Group Behavior: Interest or experience in modeling group conversation, deliberation, focus groups, juries, committees, polarization, collective decision-making, or social influence.
Experience in LLM evals, applied ML research, data science, research engineering, human data, market research, UXR, polling, behavioral science, computational social science, or behavioral economics.
Recent graduate or self-directed builder with strong evaluation, statistics, and AI tool skills.

Responsibilities

Build the measurement layer for behavioral simulation, designing evals, metrics, rubrics, datasets, dashboards, and workflows to measure the accuracy of Simile’s models in predicting human behavior across various use cases, populations, question types, and decision contexts.
Partner with the modeling team to improve models by evaluating new versions, diagnosing regressions, identifying areas for improvement, and maintaining stable eval suites that align with customer needs.
Contribute to product and applied evaluations by building evals for qualitative responses, retrieval, survey generation, AI-generated research reports, and other product surfaces where model quality impacts customer trust.
Translate subjective quality concerns into concrete rubrics, labeled data, automated graders, release criteria, and model-improvement signals.
Develop rigorous methods to compare simulated responses against human data, customer studies, Simile-collected ground truth, and behavioral datasets, addressing sampling error, uncertainty, calibration, margin of error, representativeness, and the definition of 'ground truth' for inherently noisy human behavior.
Automate evaluation workflows using modern agentic coding tools to build internal tools, inspect model outputs, create labeling workflows, validate evals, and transform fuzzy evaluation questions into functional systems.
Prototype methods for evaluating behavioral predictions using diverse data sources, including transaction or purchase behavior, product interactions, intervention response, first-party experiments, and multi-agent group settings, to help define the future of behavioral simulation evaluations.