Evaluations Engineering - Member of Technical Staff

Simile • San Francisco, CA

19h • $200,000 - $400,000

About The Position

Simile is building the first AI simulation of society, populated by generative agents based on real humans. We have pioneered the field of AI-based simulation and are developing a Foundation Model to predict human behavior. The Evaluation Engineering team at Simile faces unusual challenges in evaluating models that predict distributions of human behavior, where ground truth can be noisy and heterogeneous. This role involves partnering closely with Evals, Modeling, Product Engineering, and Data Operations to create systems that are reproducible, scalable, and useful for model development and business decisions.

As a Member of Technical Staff in Evals Engineering, you will build the systems that enable Simile to evaluate the accuracy, trustworthiness, and improvement of our human behavior simulations. Your work will span data and evaluation infrastructure, execution workflows, backend services, automation, and internal tooling. Initial focus areas include streamlining evaluation execution, strengthening evaluation versioning, data models, access controls, and automating customer validations, survey operations, and human data workflows.

Requirements

• Strong Engineering Fundamentals: Several years of experience building and maintaining production-quality software, with sound judgment in system design, testing, debugging, and maintainability.
• Data and Systems Experience: Experience building backend services, data pipelines, automation workflows, and relational data models.
• End-to-End Execution: Ability to work across data, backend, and interface layers and take ambiguous projects from technical design through deployment and adoption.
• Evaluation Judgment: Strong intuition for what makes evaluation infrastructure reliable, including versioning, provenance, reproducibility, holdout integrity, noisy ground truth, and meaningful model comparisons.
• ML and LLM Fluency: Familiarity with modern model-development and evaluation workflows sufficient to partner effectively with modeling and evaluation researchers.
• Product and User Judgment: Ability to build clear, efficient tools for researchers, engineers, data operators, and other expert users.
• Ownership and Communication: A track record of independently driving important technical work and collaborating effectively across engineering, research, and operations.

Nice To Haves

• Model-Evaluation Infrastructure: Experience building LLM or ML evaluation systems, benchmark platforms, regression suites, experiment-tracking tools, or model-quality dashboards.
• Research and Internal Tools: Experience developing technical surfaces for ML engineers, researchers, data scientists, or operations teams.
• Human Data Systems: Experience with labeling platforms, expert-review workflows, LLM-as-judge systems, grader calibration, or other human-in-the-loop evaluation methods.
• Data-Collection Automation: Experience automating surveys, experiments, customer-data ingestion, or other human data collection workflows.
• Statistical Fluency: Comfort reasoning about sampling error, uncertainty, calibration, confidence intervals, and distributional metrics.
• Sensitive Data and Access Controls: Experience designing permissions, auditability, and data-governance systems for human or customer data.
• Agentic Engineering: Experience using modern AI coding tools to accelerate development while independently testing and validating their output.

Responsibilities

• Build evaluation execution infrastructure: Develop the services, pipelines, and orchestration needed to run evaluations efficiently across datasets, model versions, populations, and use cases.
• Strengthen evaluation data systems: Design relational schemas, versioning, provenance, permissions, and quality controls that make evaluation results reproducible and trustworthy.
• Automate validation and data collection: Partner with Evals and Data Operations to streamline customer validations, survey deployment, response ingestion, and the integration of new ground truth.
• Build human data workflows: Create labeling and review tools that enable external experts and operators to contribute high-quality judgments to evaluation campaigns.
• Develop evaluation tooling: Build interfaces that help teams manage evals, compare models, investigate results, and identify regressions.

Benefits

• Competitive compensation packages that include base salary, equity, and comprehensive benefits.
• Equity: Grants are available for eligible roles, subject to board approval.
• Health & Wellness: Comprehensive medical, dental, and vision coverage.
• Time Off: Flexible time off policies to support work-life balance.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed