Director, Research - AI Evals

Figma•San Francisco, CA

1d•$258,000 - $348,000•Remote

About The Position

Figma is growing its team and is looking for a Director, Research - AI Evals to lead the measurement of quality for Figma's AI-powered experiences. This role is crucial as Figma expands its AI capabilities, focusing on defining, measuring, and rigorously assessing the quality of these features. The successful candidate will establish frameworks, quality standards, and provide trusted data to product teams for AI feature development and shipping decisions. This position involves close collaboration with Product, Design, Engineering, and Data Science teams, influencing AI shipping decisions that impact millions of users.

Requirements

10+ years of experience in product, research, applied research, or a closely related field.
2+ years of management experience.
Direct, hands-on experience owning the evaluation of AI/LLM-powered products.
Expertise in designing and running AI evaluation, including human evaluation programs, rubric and benchmark/golden-dataset construction, and inter-rater reliability.
Sound judgment on when and how to apply automated/model-based approaches (e.g., LLM-as-judge), including understanding their limitations.
Strength across both qualitative and quantitative methods, comfort with data and metrics, and the ability to reason about model behavior.
Demonstrated success in identifying, prioritizing, and evaluating risky assumptions behind ambiguous quality questions.
Proven track record of gaining buy-in from executive and cross-disciplinary stakeholders, articulating user stories and the 'so what' to inspire action.

Nice To Haves

Experience building or co-building automated evaluation pipelines and regression testing in partnership with engineering.
Familiarity with eval tooling (e.g., Braintrust, LangSmith, DeepEval, or equivalents).
Experience standing up a new function, practice, or discipline from scratch.
2+ years in product design, user-centric product management, data science, product development, and/or front-end engineering.
Familiarity and depth of experience using Figma's products.

Responsibilities

Own AI evaluation methods and operations for Figma's AI-powered experiences, defining quality dimensions, measurement strategies, and translating results into decision-ready signals.
Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, integrating human evaluation with automated/model-based approaches.
Partner with engineering to establish repeatable and reproducible evaluation pipelines and regression testing for AI features.
Produce clear readouts and dashboards to support stakeholder go/no-go and prioritization decisions.
Socialize a unified definition of quality across teams and advocate for evaluation as a strategic partner in the product process.
Manage a small team to execute AI evaluations in collaboration with contractors, internal staff, and/or LLMs.