Principal Applied Scientist for Copilot Evals

Microsoft • Redmond, WA

32d • $142,800 - $304,200

About The Position

CADET (Customer and Analytics Driven Evals Team) is building a customer-grounded quality system for Copilot. The team identifies important customer scenarios, represents them in evaluation and learning assets, runs continuous quality gates, and turns failures into reusable product and model improvements. It combines product signals such as DSAT with deep customer engagement to create representative evaluations and connect customer-grounded quality issues with quality teams.

The Principal Applied Scientist will work directly with enterprise customers and Copilot teams to translate high-value workflows, expected outcomes, and recurring pain points into trusted evaluation and learning signals. The role sets the scientific direction for customer-grounded quality, defines quality criteria, assesses whether evaluation portfolios represent real needs, diagnoses model and agent failures, and turns evidence into reusable evaluations, RLEs, reward signals, and post-training priorities. The successful candidate brings scientific depth and product judgment, translating ambiguous customer problems into rigorous, scalable methods in partnership with applied researchers and ML engineers.

Requirements

• Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field and 6+ years of related experience, or a master's degree in one of these fields and 4+ years of related experience, or a doctorate in one of these fields and 3+ years of related experience, or equivalent experience.
• Experience applying machine learning, natural language processing, information retrieval, reinforcement learning, experimentation, or evaluation methods to complex production systems.
• Hands-on ability to inspect model outputs, identify behavioral patterns, and translate qualitative judgments into testable hypotheses and measurable evaluation criteria.
• Solid understanding of statistical inference, sampling, measurement validity, bias, uncertainty, and experimental design.
• Solid written and verbal communication skills and demonstrated ability to drive results across organizational boundaries by aligning science, engineering, product, and platform teams around shared quality goals, explicit ownership boundaries, and measurable outcomes.

Nice To Haves

• Advanced degree in computer science, machine learning, statistics, applied mathematics, or a related quantitative field, or equivalent practical experience.
• Experience designing evaluations, metrics, experiments, datasets, graders, or reward functions for AI or agentic systems.
• Experience with large language models, copilots, agents, tool use, retrieval-augmented generation, or enterprise grounding.
• Experience running or partnering on RLHF, direct preference optimization, instruction tuning, fine-tuning, or human-preference data programs.
• Experience connecting offline metrics with online product behavior and customer outcomes.
• Experience working directly with enterprise customers or translating qualitative research and customer signals into scientific assets.
• Publication record, patents, or demonstrated industry impact in relevant applied research areas.

Responsibilities

• Set the scientific strategy for customer-grounded quality across priority Copilot intents, defining what good means and the tradeoffs across quality and safety attributes.
• Translate user research, enterprise customer feedback, DSAT, and production incidents into evaluation and post-training priorities, and lead cross-team creation of reusable evaluation, regression, RLE, and post-training assets for high-value workflows and failure patterns.
• Develop methods to assess evaluation-set representativeness, coverage, freshness, discrimination, grader reliability, and alignment with production outcomes; identify material gaps, drift, and emerging loss patterns.
• Design behavior and task evaluations, rubrics, graders, and calibration methods that translate qualitative customer expectations into measurable release-over-release quality.
• Establish methods to attribute quality losses across grounding, retrieval, tools, orchestration, model reasoning, response generation, and evaluation, linking offline movement with online signals such as DSAT, task completion, retries, abandonment, and escalation.
• Set a high bar for scientific rigor, reproducibility, documentation, and interpretation of results.
• Mentor scientists and engineers and influence evaluation and post-training strategy across organizational boundaries.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal