Principal Applied Scientist - Copilot Quality

Microsoft • Redmond, WA

12d • $165,600 - $331,200

About The Position

Microsoft is seeking a Principal Applied Scientist to shape the strategy, technical roadmap, and execution of quality systems for Copilot. The role focuses on making Copilot a trusted, capable, continuously improving AI assistant by turning customer signals into lasting product improvements. The scientist will set a shared quality bar across Copilot experiences and accelerate the process from customer signal through diagnosis, evaluation, improvement, and verified impact. The work spans evaluation, experimentation, production intelligence, AI safety, reliability, customer feedback, and product improvement, with machine learning at its core.

Requirements

• Bachelor’s degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or a related field and 8+ years of related experience, or a master’s degree in one of these fields and 6+ years of related experience, or a doctorate in one of these fields and 5+ years of related experience, or equivalent experience.

Nice To Haves

• Advanced degree (MS or PhD) in a relevant technical discipline.
• Experience improving large language models, AI assistants, agents, or other large-scale AI products.
• Ability to work in a fast-paced, research-driven environment while maintaining high technical standards.
• Interest in advancing human-aligned AI and staying current with emerging model-training technologies.
• Demonstrated commitment to inclusive leadership, talent development, and fostering a culture where teams can do the best work of their careers.
• Significant experience building, evaluating, or improving production machine-learning or AI systems.
• Demonstrated expertise in at least one relevant area, such as large-model evaluation, deep learning, post-training, reward modeling, reinforcement learning, retrieval, or ranking.
• Experience designing rigorous metrics or experiments and using results to guide technical and product decisions.

Responsibilities

• Develop online and offline quality metrics, representative evaluation datasets, human and AI-assisted labeling approaches, and benchmarks grounded in real customer tasks.
• Validate that quality measures are statistically sound, sensitive to meaningful changes, and predictive of customer outcomes.
• Identify high-impact failure modes across Copilot experiences and use data, evaluations, and customer feedback to determine root causes.
• Improve quality through approaches such as post-training, retrieval and ranking, prompting, orchestration, and product changes.
• Design experiments that determine whether promising offline results lead to improvements customers experience.
• Establish A/B experimentation, quality gates, regression protection, and rollout criteria connecting offline gains with customer satisfaction, task success, engagement, retention, latency, reliability, safety, and cost.
• Create operating mechanisms to rapidly resolve quality tradeoffs, dependencies, and release risks.
• Shape Copilot quality strategy and technical direction and lead high-impact teams across relevant quality areas.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal