Researcher, AI Evaluations
About The Position
Figma is hiring a senior individual contributor Researcher, AI Evaluations to join AI Insights within its Research team. The role strengthens model, agent system, and evaluation work through hands-on studies, clear rubrics, and reliable measures of quality that reflect user needs. Findings will guide model and product improvements. The position has no direct reports, reports to the AI Insights lead, and works directly with AI Research within AI Platform. It may be based at a US hub or performed remotely within the United States. The researcher owns evaluation research for priority AI Research workstreams, defining quality criteria, designing studies, analyzing results, and recommending actions. They develop rubrics, representative evaluation datasets, benchmarks, and quality thresholds grounded in user needs and expert design judgment. In partnership with AI Research and Modeling Platform engineers, they integrate validated methods into repeatable evaluations and regression tests. They explain quality, failure patterns, and uncertainty to inform model selection and improvements, product priorities, and release decisions. They also validate automated graders against human ratings, including LLM-as-judge methods, and connect user research with model development by sharing reusable methods, reviewing work, and mentoring peers.
Requirements
- 7+ years of experience in research, applied research, or a related field.
- A proven track record of independently leading complex evaluation projects and delivering measurable impact.
- Hands-on experience evaluating AI products and translating evaluation findings into meaningful model and product improvements.
- Expertise designing human evaluations, scoring rubrics, datasets, and benchmarks, including measuring inter-rater agreement and validating automated graders.
- Strong qualitative and quantitative research skills, including analyzing data, interpreting uncertainty, and drawing actionable insights into model behavior and performance.
- Excellent problem-solving and stakeholder management skills; ability to navigate ambiguity, prioritize risks, balance speed with rigor, and deliver recommendations that influence cross-functional decisions.
Nice To Haves
- Experience integrating automated evaluations and regression testing into engineering workflows; familiarity with Braintrust, LangSmith, DeepEval, or equivalent platforms.
- Proven ability to develop scalable, reusable evaluation frameworks and drive adoption of consistent quality standards across teams.
- Background in product design, product management, data science, or engineering, with familiarity in Figma or similar design and developer tools.
- Strong statistical expertise in sampling, significance testing, and agreement metrics, with experience conducting experiments or analyzing performance metrics for AI-powered features.
- Advanced training in a quantitative or behavioral discipline through formal education or equivalent hands-on experience.
Responsibilities
- Own evaluation research for priority AI Research workstreams: define quality criteria, design studies, analyze results, and recommend actions.
- Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds that reflect user needs and expert design judgment.
- Work with AI Research and Modeling Platform engineers to integrate validated methods into repeatable evaluations and regression tests.
- Explain quality, failure patterns, and uncertainty, and recommend actions to guide model selection, model improvements, product priorities, and release decisions.
- Validate automated graders against human ratings, including LLM-as-judge methods; identify bias, gaps, and uncertainty.
- Connect user research with AI Research’s model development work by sharing reusable methods, reviewing work, and mentoring peers.
Benefits
- Equity
- Health coverage
- Dental coverage
- Vision coverage
- Retirement benefits with company contributions
- Parental leave
- Reproductive or family planning support
- Mental health and wellness benefits
- Paid time off, including paid sick leave, holidays, and other leave benefits
- Employer-provided paid flexible PTO for exempt employees, in addition to flexible paid sick leave
- Company recharge days
- Cell phone and home internet reimbursements
- Lifestyle spending accounts
- Annual bonus plan for eligible non-sales roles
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
