Senior Technical Program Manager, Copilot

Microsoft•Mountain View, CA

About The Position

Own the Model Feedback Lifecycle: Design and manage processes for collecting, triaging, and analyzing feedback from production usage. Build dashboards and reporting systems to surface insights and drive data-informed improvements to Copilot, addressing customer needs with high quality and efficiency. Identify and mitigate project risks to ensure successful outcomes. Cross-Team Coordination: Partner with Research, Engineering, Data, and Product teams to align on goals, identify opportunities for enhancing model performance, share learnings, and drive execution at scale. Vendor Management: Oversee external vendors for data labeling and human evaluation programs. Quality Frameworks: Build and run systems to assess and enhance model responses, such as annotation platforms and evaluation pipelines. Review performance metrics to find areas for improvement. Data-Driven Decision Making: Collaborate with analytics teams to define success metrics and correlate improvements to user impact. Operational Excellence: Manage all components of project planning, execution, and delivery by developing consistent processes that accommodate product complexity and organizational growth, ensuring coordination with overall product strategy and deadlines. Strategic Program Architecture: Partner with other TPgMs and leadership to design scalable processes and frameworks that extend beyond model response quality, supporting evolving priorities across Copilot. Maintain a deep understanding of the latest advancements in AI and Machine Learning.

Requirements

Bachelor's Degree AND 4+ years of experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
2+ year(s) of experience managing cross-functional and/or cross-team projects.
2+ year(s) of experience leading AI-focused products or programs with direct involvement in model evaluation strategies (Evals), optimization through loss function analysis (Losses), or overseeing data collection pipelines to support scalable training and validation workflows (Data Collection).
2+ year(s) of experience reading and/or writing code (e.g., sample documentation, product demos).
Bachelor's Degree AND 8+ years of experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
6+ years of experience managing cross-functional and/or cross-team projects.

Nice To Haves

Prior exposure to Large Language Models (LLMs), model evaluation, or human-in-the-loop systems.
Experience working in a startup-like environment.

Responsibilities

Design and manage processes for collecting, triaging, and analyzing feedback from production usage.
Build dashboards and reporting systems to surface insights and drive data-informed improvements to Copilot.
Identify and mitigate project risks.
Partner with Research, Engineering, Data, and Product teams to align on goals.
Oversee external vendors for data labeling and human evaluation programs.
Build and run systems to assess and enhance model responses, such as annotation platforms and evaluation pipelines.
Collaborate with analytics teams to define success metrics and correlate improvements to user impact.
Manage all components of project planning, execution, and delivery.
Partner with other TPgMs and leadership to design scalable processes and frameworks.
Maintain a deep understanding of the latest advancements in AI and Machine Learning.