Relevance Metrics Data or Applied Scientist

Microsoft • Redmond, WA

13h • Hybrid

About The Position

We are seeking an accomplished Principal Data Scientist to lead the development and evaluation of offline search metrics for Bing. This position is integral to the advancement of search quality measurement, driving data-driven insights through rigorous offline analyses. As a member of the Bing Metrics team, you will leverage state-of-the-art BigData technologies to sample, process, and analyze vast search logs, formulate and validate hypotheses regarding search relevance, and deliver actionable findings to leadership. This role offers the opportunity to collaborate extensively across engineering, product, and research teams, shaping the future of search quality at scale.
As Principal Data Scientist, you will directly influence Bing’s search quality and relevance, drive innovation in offline metric development, and enable data-informed decision-making at the highest levels. Your work will shape product strategy and have a broad impact across the Bing organization, fostering collaboration and continuous improvement in search experiences worldwide.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Starting January 26, 2026, Microsoft AI (MAI) employees who live within a 50- mile commute of a designated Microsoft office in the U.S. or 25-mile commute of a non-U.S., country-specific location are expected to work from the office at least four days per week. This expectation is subject to local law and may vary by jurisdiction.

Requirements

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR equivalent experience.

Nice To Haves

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 8+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 12+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR equivalent experience.
• Extensive experience in Information Retrieval, including designing, evaluating, and optimizing search and ranking algorithms.
• Demonstrated ability to develop and validate metrics for Information Retrieval systems, ensuring robust measurement of relevance, accuracy, and user satisfaction.
• Expertise in crowdscience methodologies, including designing and running large-scale crowdsourcing experiments for data annotation and model evaluation.
• Proven track record in architecting and deploying large-scale data pipelines for real-time and batch processing of heterogeneous data sources.
• Solid background in experimental design, statistical analysis, and A/B testing for data-driven product improvements.
• Ability to lead cross-functional teams and mentor junior scientists in best practices for data science and machine learning.
• Ability to work independently, solid collaboration and communication skills
• Familiarity with Python, T-SQL, (HTML/JS for dashboarding).

Responsibilities

• Sample large, representative datasets from extensive search log repositories utilizing BigData Map-Reduce frameworks and distributed data platforms.
• Architect and implement scalable data processing and analysis pipelines for offline metric computation, leveraging modern data engineering best practices.
• Formulate, test, and validate hypotheses regarding search result quality using advanced statistical methods and machine learning models.
• Design labeling protocols and manage trained crowd judges and auditors for high-fidelity data annotation and validation of search results.
• Adapt traditional evaluation workflows to incorporate LLM-as-a-judge and fine-tuned LLMs, ensuring robust and scalable quality assessments.
• Develop and deliver custom reports, visualizations, and presentations to communicate insights and recommendations to senior leadership.
• Collaborate across multidisciplinary teams to extend offline metric methodologies and support innovative search experiences.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level