Principal Technical Program Manager
About The Position
Microsoft Search is developing AI-powered experiences that help users find information, accomplish tasks, and make decisions. This strategic role will lead the strategy and execution of Bing's offline quality measurement ecosystem, spanning product management, evaluation science, data science, user research, and engineering. The role defines how search quality, relevance, presentation, and competitive performance are measured across Bing and Copilot Search experiences. Its metrics and evaluation systems inform product investment decisions, experimentation, model launches, and executive scorecards across Microsoft AI Experiences. The position has broad organizational impact and significant executive visibility. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. Employees are expected to bring a growth mindset, innovate, collaborate, and uphold respect, integrity, accountability, and inclusion. Starting January 26, 2026, Microsoft AI employees living within a 50-mile commute of a designated U.S. Microsoft office, or within a 25-mile commute of a designated non-U.S. location, are expected to work from the office at least four days per week, subject to local law and jurisdictional variation.
Requirements
- Bachelor's degree and 6+ years of experience in engineering, product/technical program management, data analysis, or product development, or equivalent experience.
- 3+ years of experience managing cross-functional and/or cross-team projects.
Nice To Haves
- Bachelor's degree and 12+ years of experience in engineering, product/technical program management, data analysis, or product development, or equivalent experience.
- Experience defining or operating metrics used for large-scale product decisions.
- Demonstrated ability to lead cross-functional initiatives across multiple teams.
- Strong analytical and quantitative problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Experience in search, recommendation systems, ranking, AI products, or large-scale consumer experiences.
- Experience building evaluation frameworks for LLMs, retrieval systems, or generative AI applications.
- Familiarity with offline evaluation methodologies and human rating systems.
- Experience partnering with Data Science, Applied Science, and Research organizations.
- Proven success driving strategy in ambiguous and rapidly evolving technical domains.
Responsibilities
- Own the long-term vision and roadmap for Bing's offline quality metrics ecosystem.
- Define measurement strategies for AI-powered search, answer experiences, ranking systems, retrieval, multimodal content, and agentic experiences.
- Identify gaps in existing quality measurements and drive the development of next-generation metrics.
- Partner with Data Scientists, Applied Scientists, and Research teams to create robust, trustworthy, and actionable quality metrics.
- Drive metric validation, calibration, and governance frameworks.
- Ensure offline metrics predict user value and align with online outcomes.
- Drive alignment across Search, Copilot, Answer Experiences, Ranking, Ads, Multimedia, and other partner teams.
- Influence product roadmaps through measurement-based insights.
- Serve as the primary point of contact for quality measurement across multiple organizations.
- Build diagnostic frameworks to explain score movement and surface scenarios, segments, and page configurations driving competitive gaps; translate findings into investment opportunities.
- Enable teams to use measurement systems for product development, experiment evaluation, launch readiness, executive scorecards, competitive tracking, and quality monitoring.
- Translate complex evaluation concepts into actionable recommendations for engineering and leadership teams.
- Stay current with advances in search evaluation, LLM evaluation, AI quality measurement, human-in-the-loop evaluation, and synthetic evaluation approaches.
- Influence Microsoft's overall approach to AI quality measurement.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
