Principal Applied Scientist - Copilot Quality

Microsoft•Redmond, WA

12d

About The Position

Microsoft Copilot is used by millions of people every day. The Copilot organization aims to make it a trusted, capable, and continuously improving AI assistant by converting customer signals into lasting product improvements. The Principal Applied Scientist will define and drive the strategy, technical roadmap, and execution for Copilot quality systems, helping establish a shared quality bar across Copilot experiences and accelerating the loop from customer signal to diagnosis, evaluation, improvement, and verified product impact. The role leads high-impact work across evaluation, experimentation, production intelligence, AI safety, reliability, customer feedback, and product improvement. It involves building reliable measures and representative evaluations, identifying and diagnosing failure modes, guiding quality improvements, and ensuring changes deliver safe, reliable improvements for customers.

Requirements

Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field and 8+ years of related experience; or a master's degree in one of these fields and 6+ years of related experience; or a doctorate in one of these fields and 5+ years of related experience; or equivalent experience.
Related experience may include statistics, predictive analytics, or research.

Nice To Haves

Advanced degree (MS or PhD) in a relevant technical discipline.
Experience improving large language models, AI assistants, agents, or other large-scale AI products.
Ability to operate in a fast-paced, research-driven environment while maintaining high technical standards.
Passion for advancing human-aligned AI and staying at the forefront of emerging model-training technologies.
Demonstrated commitment to inclusive leadership, talent development, and fostering a culture where teams do the best work of their careers.
Significant experience building, evaluating, or improving production ML or AI systems.
Demonstrated expertise in at least one relevant area, such as large-model evaluation, deep learning, post-training, reward modeling, reinforcement learning, retrieval, or ranking.
Experience designing rigorous metrics or experiments and using results to guide technical and product decisions.

Responsibilities

Develop online and offline quality metrics, representative evaluation datasets, human and AI-assisted labeling approaches, and benchmarks grounded in real customer tasks.
Validate that quality measures are statistically sound, sensitive to meaningful changes, and predictive of customer outcomes.
Identify high-impact failure modes across Copilot experiences and use data, evaluations, and customer feedback to determine root causes.
Improve quality through approaches such as post-training, retrieval and ranking, prompting, orchestration, and product changes.
Design experiments to distinguish promising offline results from improvements customers actually experience.
Establish rigorous A/B experimentation, quality gates, regression protection, and rollout criteria connecting offline gains with customer satisfaction, task success, engagement, retention, latency, reliability, safety, and cost.
Create operating mechanisms to rapidly resolve quality tradeoffs, dependencies, and release risks.
Shape technical and product direction and lead high-impact teams working across Copilot quality disciplines.