Principal Product Manager - Copilot Extensibility DevX

Microsoft • Redmond, WA

3d • $142,800 - $304,200

About The Position

Microsoft 365 is seeking a Principal Product Manager to shape how builders prove, understand, and continuously improve the quality and value of Copilot plugins. The role owns the strategy and roadmap for customer-facing plugin evaluations and quality measurement, giving builders and owners evidence that plugins work, meet their intended quality bar, deliver value, and provide actionable guidance for improvement.

The position defines an adaptive strategy for evaluation experiences and quality measurement across the plugin lifecycle, from pre-production evaluation through continuous improvement. In partnership with the plugin DevX architect and the Observability and Analytics product leader, the Product Manager connects evaluation findings with production signals to create an end-to-end view of plugin value and a shared process for improving, scaling, consolidating, or retiring plugins.

The Principal Product Manager translates ambiguous developer and platform needs into durable product direction, leads complex evaluation and quality-measurement decisions, and aligns engineering, applied science, design, security, compliance, and partner teams. The role defines evaluation contracts and feedback loops that connect pre-production evidence with the developer journey and production signals, enabling continuous, evidence-driven improvement across Microsoft’s Copilot ecosystem.

Requirements

• Bachelor's Degree and 8+ years of experience in product/service/program management or software development, or equivalent experience.
• Experience with AI or LLM evaluation, including nondeterministic outputs, LLM-as-judge methods, human-rating calibration, multi-step agent or tool-use evaluation, dataset development, or offline-to-online measurement.
• Experience defining product strategy and leading developer platforms, extensibility systems, AI products, or evaluation capabilities from concept through launch and sustained adoption.
• Demonstrated data-driven product management using evaluation findings, telemetry, customer feedback, adoption, or business-value signals to prioritize investments and measure outcomes.
• Experience partnering with engineering on platform architecture, APIs, SDKs, extensibility models, data systems, or production services.
• Hands-on software development or equivalent experience building prototypes and integrations using APIs, SDKs, or AI-assisted development tools.
• Experience building products for technical audiences and translating developer needs into clear requirements, roadmaps, and measurable outcomes.
• Experience influencing senior product and engineering stakeholders and leading complex technical initiatives across organizations without direct authority.

Nice To Haves

• Bachelor’s Degree and 12+ years of experience in product management, service management, program management, software development, or a related field, or equivalent experience.

Responsibilities

• Own the product vision, strategy, investment priorities, and roadmap for Copilot plugin evaluation and quality measurement.
• Establish quality and Responsible AI models, regression protections, and release criteria for groundedness, prompt injection, harmful content, reliability, and performance.
• Build evaluation into CLI and IDE workflows, CI/CD, and publishing so developers can create evaluation sets, interpret results, identify gaps, and improve plugins.
• Connect evaluation findings with production telemetry, adoption, support, customer, and business-value signals through a shared scorecard that guides investments.
• Validate product concepts through AI-assisted prototypes and translate them into architecture, APIs, reusable components, integration contracts, and clear engineering handoffs.
• Represent the strategy and lead cross-organizational decisions, balancing developer value, platform reuse, responsible delivery, cost, and speed.
• Define evaluation methods for nondeterministic outputs, multi-step agent and tool-use flows, LLM-as-judge scoring calibrated against human ratings, evaluation datasets, and offline-versus-online gaps.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal