Data Scientist, EngThrive Insights - CoreAI

Microsoft•Redmond, WA

1d•$102,100 - $219,200

About The Position

The Data Scientist will join Microsoft’s Engineering Thrive Insights team, which develops the data platform, metrics, and insights used to measure and improve engineering outcomes, including developer satisfaction, change velocity, build quality, and AI tool impact. The role focuses on the AI Data Flywheel, a platform that collects signals from engineers using AI coding tools and turns them into actionable insights and experiments for product improvement. The position defines success metrics, designs and analyzes experiments, develops statistical and machine-learning models, and identifies opportunities to improve AI-assisted software development. It builds measurement frameworks, evaluates product changes, and translates behavioral and product data into recommendations for engineering and product teams. The work also involves experimentation, causal inference, developer productivity frameworks such as DORA and SPACE, AI product evaluation, and privacy-conscious analytics. The role evaluates AI and LLM systems, prototypes and operationalizes models and applications, supports retrieval-augmented generation workflows, prepares and analyzes data, and develops models that can run at scale in partnership with data engineering teams. It also maintains awareness of industry developments and contributes to thought leadership and intellectual property best practices.

Requirements

Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or a related field; or a master’s degree in one of these fields plus at least 1 year of data-science or consulting experience; or a bachelor’s degree in one of these fields plus at least 2 years of data-science experience; or equivalent experience.
Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

Doctorate in a listed quantitative or computing field plus at least 1 year of data-science experience; or a master’s degree plus at least 3 years; or a bachelor’s degree plus at least 5 years; or equivalent experience.
Experience with observational causal inference, controlled-experiment design, regression modeling, and hypothesis testing.
Experience evaluating LLMs and/or AI agents, including designing offline or online evaluations, defining quality metrics, and measuring model behavior.
Experience designing surveys and analyzing response data.
Proficiency in Python.
Proficiency in analytical query languages such as SQL or KQL.
Experience developing analytics on Spark-based big data platforms such as Databricks.
Ability to craft and present compelling data stories to business decision-makers.
Familiarity with developer productivity metrics and frameworks such as DORA and SPACE.
Demonstrated effective use of AI coding assistants in a daily data-science development workflow.

Responsibilities

Examine projects using data science and business understanding, considering factors that may influence outcomes and evaluating plans for resources, risks, contingencies, requirements, assumptions, and constraints.
Document business objectives, communicate them in analytical and technical terms, and share insights with stakeholders.
Collaborate with team members to define metrics and frameworks for evaluating AI/LLM systems and agentic product performance.
Experiment with models and prompting approaches, diagnose basic AI/LLM issues, and identify opportunities for improvement.
Prototype and develop AI models in containerized environments, such as Azure AI Foundry and LangChain, to build, validate, and operationalize scalable tools and applications.
Develop and deploy basic retrieval-augmented generation systems, helping select data sources, retrieval systems, and generative models.
Assist senior team members in assessing and refining LLM systems and models.
Identify and acquire project data; use querying, visualization, and reporting to describe its properties and explore important attributes.
Contribute to data-quality reports describing results, initial findings, and project impact, using appropriate AI tools such as Codex and GitHub Copilot.
Collaborate on data-science experiments using established methodologies, statistics, optimization, probability theory, and appropriate software and tools.
Partner with Solution Architects, Consultants, and Data Engineers on data preparation, while following Microsoft privacy policy and identifying data-integrity problems.
Select appropriate statistical, causal, and machine-learning approaches; prepare data, train and optimize models, and evaluate outputs for statistical and business significance.
Account for data leakage, bias-variance tradeoffs, and methodological limitations in modeling work.
Write scripts in appropriate languages, including T-SQL, U-SQL, KQL, Python, or R.
Form hypotheses, design controlled experiments, analyze results with statistical tests, and communicate findings and data-quality issues to business stakeholders and diverse audiences.
Address model deployment considerations such as performance, scalability, monitoring, maintenance, production integration, and stability; partner with data engineering teams to develop operational models that run at scale.
Keep current with embedding models, vector databases, industry tools and practices, and discipline trends; attend internal research conferences and training when appropriate.
Contribute to thought leadership and intellectual-property best practices.