Researcher, AI Evaluations

Figma•San Francisco, CA

1d•$169,000 - $250,000

About The Position

Figma is hiring a senior individual contributor Researcher, AI Evaluations to join AI Insights within its Research team. The role has no direct reports, reports to the AI Insights lead, and works directly with AI Research within AI Platform. The researcher will strengthen model, agent system, and evaluation work through hands-on studies, clear rubrics, and reliable measures of quality that reflect user needs. Findings will guide model and product improvements. The position can be held at one of Figma’s US hubs or remotely in the United States. It is full-time.

Requirements

At least 7 years of experience in research, applied research, or a related field.
A proven track record of independently leading complex evaluation projects and delivering measurable impact.
Hands-on experience evaluating AI products and translating evaluation findings into meaningful model and product improvements.
Expertise in designing human evaluations, scoring rubrics, datasets, and benchmarks, including measuring inter-rater agreement and validating automated graders.
Strong qualitative and quantitative research skills, including analyzing data, interpreting uncertainty, and drawing actionable insights into model behavior and performance.
Excellent problem-solving and stakeholder management skills, including navigating ambiguity, prioritizing risks, balancing speed with rigor, and delivering recommendations that influence cross-functional decisions.

Nice To Haves

Experience integrating automated evaluations and regression testing into engineering workflows, with familiarity with Braintrust, LangSmith, DeepEval, or equivalent platforms.
Ability to develop scalable, reusable evaluation frameworks and drive adoption of consistent quality standards across teams.
Background in product design, product management, data science, or engineering, with familiarity in Figma or similar design and developer tools.
Strong statistical expertise in sampling, significance testing, and agreement metrics, with experience conducting experiments or analyzing performance metrics for AI-powered features.
Advanced training in a quantitative or behavioral discipline, through formal education or equivalent hands-on experience.

Responsibilities

Own evaluation research for priority AI Research workstreams by defining quality criteria, designing studies, analyzing results, and recommending actions.
Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds that reflect user needs and expert design judgment.
Work with AI Research and Modeling Platform engineers to integrate validated methods into repeatable evaluations and regression tests.
Explain quality, failure patterns, and uncertainty, and recommend actions that guide model selection, model improvements, product priorities, and release decisions.
Validate automated graders against human ratings, including LLM-as-judge methods, and identify bias, gaps, and uncertainty.
Connect user research with AI Research’s model development work, share reusable methods, review work, and mentor peers.