Data Scientist, EngThrive Insights - CoreAI

Microsoft•Redmond, WA

About The Position

The Data Scientist will join Microsoft’s Engineering Thrive Insights team, which develops the data platform, metrics, and insights used to measure and improve engineering, including developer satisfaction, change velocity, build quality, and AI tool impact. The role’s primary focus is the AI Data Flywheel: a platform that collects signals from engineers using AI coding tools and turns them into actionable insights and experiments for continuous product improvement. The position defines success metrics, designs and analyzes experiments, develops statistical and machine-learning models, and identifies opportunities to improve AI-assisted software development. It builds measurement frameworks, evaluates product changes, and translates behavioral and product data into recommendations for engineering and product teams. The role also develops expertise in experimentation, causal inference, developer productivity frameworks such as DORA and SPACE, AI product evaluation, and privacy-conscious analytics. The Data Scientist works with AI and LLM systems, data sources, retrieval systems, and generative models; develops scalable analytical and operational solutions; and partners with data engineering and other teams. The position requires compliance with Microsoft privacy policies and the ability to meet Microsoft, customer, and/or government security screening requirements, including a Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Requirements

Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or a related field; or a master’s degree in one of these fields and at least 1 year of data-science or consulting experience; or a bachelor’s degree in one of these fields and at least 2 years of data-science experience; or equivalent experience.
Data-science experience may include managing structured and unstructured data, applying statistical techniques, and reporting results.
Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

Doctorate in a relevant field and at least 1 year of data-science experience; or a master’s degree in a relevant field and at least 3 years of data-science experience; or a bachelor’s degree in a relevant field and at least 5 years of data-science experience; or equivalent experience.
Experience with observational causal-inference methods, controlled-experiment design, regression modeling, and hypothesis testing.
Experience evaluating LLMs and/or AI agents, including designing offline and online evaluations, defining quality metrics, and measuring model behavior.
Experience designing surveys and analyzing response data.
Proficiency in Python programming.
Proficiency in analytical query languages such as SQL, KQL, or similar.
Experience developing analytics on Spark-based big-data platforms such as Databricks.
Ability to craft and present compelling data stories to business decision-makers.
Familiarity with developer productivity metrics and frameworks such as DORA and SPACE.
Demonstrated effective use of AI coding assistants in a daily data-science development workflow.

Responsibilities

Apply data-science and business knowledge to examine projects and identify factors that may influence outcomes within a technical area.
Evaluate project plans for resources, risks, contingencies, requirements, assumptions, and constraints; document business objectives and communicate them in analytical and technical terms.
Share insights with stakeholders and collaborate with team members to define metrics and frameworks for evaluating AI/LLM systems and agentic product performance.
Experiment with models and approaches, such as LLMs and prompting strategies, before selection for product or solution tasks.
Diagnose basic issues in AI/LLM systems and identify opportunities for improvement.
Prototype and develop AI models in containerized environments, such as Azure AI Foundry or LangChain, to build, operationalize, and validate scalable tools, applications, and solutions.
Develop and deploy basic retrieval-augmented generation workflows, helping select data sources, retrieval systems, and generative models to support quality outputs.
Assist senior team members in assessing and refining LLM systems and models.
Identify data sources needed for project plans; use querying, visualization, and reporting techniques to describe acquired data and its properties.
Explore data for key attributes and contribute to data-quality reports covering results, initial findings, and project impact, using appropriate AI tools such as Codex and GitHub Copilot.
Conduct data-science experiments with established methodologies, statistics, optimization, probability theory, and appropriate software and statistical packages.
Assess and select appropriate tools and techniques, and partner with Solution Architects, Consultants, and Data Engineers on data preparation.
Follow Microsoft privacy policies when collecting and preparing data and using AI tools.
Identify data-integrity problems and apply modeling and machine-learning knowledge, including statistical analysis and causal modeling, to select suitable approaches.
Select methods for data preparation, model training and optimization, and output evaluation, accounting for statistical and business significance, data leakage, bias-variance tradeoffs, and methodological limitations.
Write scripts in appropriate languages, including T-SQL, U-SQL, KQL, Python, and R.
Construct hypotheses, design controlled experiments, analyze results with statistical tests, and communicate findings to business stakeholders.
Communicate data-quality issues and initiatives effectively to diverse audiences.
Account for model deployment considerations such as performance, scalability, monitoring, maintenance, production-system integration, and stability.
Develop operational models that run at scale in partnership with data engineering teams.
Keep current on embedding models, vector databases, industry tools and techniques, process improvements, and trends in the discipline.
Attend internal research conferences and participate in hands-on training when appropriate.
Contribute to thought leadership and intellectual-property best practices.