Principal Applied Scientist

Microsoft

15h

About The Position

Security represents the most critical priorities for our customers in a world awash in digital threats, regulatory scrutiny, and estate complexity. Microsoft Security aspires to make the world a safer place for all. We want to reshape security and empower every user, customer, and developer with a security cloud that protects them with end to end, simplified solutions. The Microsoft Security organization accelerates Microsoft’s mission and bold ambitions to ensure that our company and industry is securing digital technology platforms, devices, and clouds in our customers’ heterogeneous environments, as well as ensuring the security of our own internal estate. Our culture is centered on embracing a growth mindset, a theme of inspiring excellence, and encouraging teams and leaders to bring their best each day. In doing so, we create life-changing innovations that impact billions of lives around the world. Our team builds and operates the end-to-end measurement stack for Microsoft Security’s AI agents. We build and collaborate with industry on security domain benchmarks for foundation model comparisons, empowering product teams to build at the frontier edge of AI while driving progress in model research and development. We build rigorous evaluations of complete agentic systems, ensuring only the best solutions reach our customers. As a Principal Applied Scientist, you will define and drive the evaluation strategy for AI security agents across multiple products and orchestration frameworks. You will help set technical direction for how we measure agent capabilities at scale. You will also influence product roadmaps through evaluation insights, partner with product and engineering to translate results into readiness gates, model selection criteria, and go/no-go decisions that enable teams to ship, safe, reliable security agents. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Requirements

Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR equivalent experience.
Candidates must be able to meet Microsoft, customer and/or government security screening requirements are required for this role.
These requirements include, but are not limited to the following specialized security screenings:
Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Nice To Haves

Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 9+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) OR equivalent experience.
Demonstrated experience shipping ML systems in production with experiment design, evaluation, and measurement methodology.
Ability to design controlled experiments and ablations to isolate system behavior
Experience in cybersecurity or security-adjacent domains (safety, abuse/fraud, or risk)
Track record of technical leadership across teams in the form of setting standards and driving alignment in ambiguity
Deep expertise across multiple security domains (detection, investigation, response, threat intelligence, automation)
Experience evaluating LLM-based systems and agentic workflows, including tool use, planning, reasoning, and robustness.
Track record of setting evaluation standards adopted across orgs, mentoring ICs in methodology, and driving executive-level decisions through rigorous experimental analysis.

Responsibilities

Technical Leadership & Ownership: Define and drive technical strategy for security agent evaluations across Microsoft Security.
Lead assessments of full agent systems, set evaluation standards adopted by multiple product groups, and translate findings into exec-ready recommendations and deployment quality gates
Agentic System Evaluation & Measurement: Define evaluation methodology for agent actions in realistic security scenarios covering planning, tool use, reasoning, robustness, and failure modes.
Establish metrics that directly influence product readiness and release decisions.
Evaluation Infrastructure & Harnesses: Architect reusable evaluation platform and harnesses across orchestration frameworks.
Enable controlled testing on production-representative data with appropriate privacy controls.
Collect trajectories, telemetry, and results at scale
Data & Signals for Evaluation: Set data standards and governance for agent evaluation datasets and grading rubrics.
Ensure reproducibility, privacy-preserving data practices, and long-term reuse across evaluation scenarios.
Research & Innovation: Collaborate with cross-functional teams to advance evaluation research, prototype novel methods, and land them into the evaluation platform used by product teams.
Contribute to publications and external partnerships where appropriate.