Senior Product Manager, Copilot AI

Microsoft • Redmond, WA

5d

About The Position

Develop and execute on LLM platform strategy for Copilot that extend language model's capabilities. Prototype approaches by steering language models to drive response quality across a wide range of scenarios. Identify and prioritize platform, orchestration and language model issues that impact quality, factuality and safety and working with engineers and researchers to find a path to resolution. Define and build measurable evaluations with relevant datasets to demonstrate quality improvements. Define, deploy and manage experiments in production that impact language model's tool use, driving measurable improvements in relevance for and engagement with Copilot users. Partner with product teams to scale tool building and work with inference, agents and orchestration teams to resolve dependencies. Accountable to own the status of key projects, proactively identifying risks and proposing solutions to ensure timely delivery. LLMs will be your clay - you should know how to prompt engineer them, how to evaluate them, and understand how they are tuned.

Requirements

• Bachelor's Degree AND 5+ years experience in product management OR equivalent experience.
• 3+ years of experience leading ambiguous product areas, defining requirements, developing roadmaps, and working with multi-disciplinary teams to execute them.
• 2+ years of experience building ML-powered or LLM-powered products.
• Hands-on experience with LLM APIs (e.g. OpenAI, Anthropic, Azure OpenAI), embeddings, vector databases, and tool use.
• Hands-on experience with prompt design, context window management, and model evaluation.
• 2+ years of hands-on experience in building retrieval and/or recommendation systems, driving measurable improvements in relevance, engagement, and conversion through data-driven experimentation.
• Skilled in translating user intent into product strategy and collaborating closely with engineering and ML teams to deliver scalable, personalized experiences.
• 1+ years of experience in building LLM applications with familiarity in agent and orchestration frameworks, tool use, LLM evaluations and driving efficiency improvements.
• Technical depth in software development, data science and machine learning.
• Experience using and applying Large Language Models.
• LLMs will be your clay - you should know how to prompt engineer them, how to evaluate them, and understand how they are tuned.

Responsibilities

• Develop and execute on LLM platform strategy for Copilot that extend language model's capabilities.
• Prototype approaches by steering language models to drive response quality across a wide range of scenarios.
• Identify and prioritize platform, orchestration and language model issues that impact quality, factuality and safety and working with engineers and researchers to find a path to resolution.
• Define and build measurable evaluations with relevant datasets to demonstrate quality improvements.
• Define, deploy and manage experiments in production that impact language model's tool use, driving measurable improvements in relevance for and engagement with Copilot users.
• Partner with product teams to scale tool building and work with inference, agents and orchestration teams to resolve dependencies.
• Accountable to own the status of key projects, proactively identifying risks and proposing solutions to ensure timely delivery.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees