Principal Applied Scientist - CoreAI Foundry Observability

Microsoft•Redmond, WA
•$165,600 - $296,400

About The Position

The Principal Applied Scientist will work in Microsoft’s CoreAI Foundry observability group, advancing scientific methods for AI agents and agent-based systems. The role focuses on defining how agent quality is measured, diagnosed, and improved across offline evaluation, online signals, and production telemetry, and translating that science into practical capabilities for Foundry customers. The position leads science for agent and multi-agent systems, including offline, online, and continuous evaluation and agent-improvement methods tied to real product behavior. It develops measurement frameworks to explain regressions, detect failure modes, and support confident product and release decisions. The scientist also partners with engineering, product, and platform teams to operationalize evaluation and observability across Foundry, Azure Monitor, and agent runtimes, while setting scientific direction, mentoring colleagues, and influencing research-informed product decisions.

Requirements

  • Bachelor's degree in Computer Science, Computational Linguistics, or a related field and 8+ years of related experience, such as statistics, predictive analytics, or research; or equivalent experience.
  • Alternatively, a master's degree in Computer Science, Computational Linguistics, or a related field and 6+ years of related experience, such as statistics, predictive analytics, or research.
  • Alternatively, a doctorate in Computer Science, Computational Linguistics, or a related field and 5+ years of related experience, such as statistics, predictive analytics, or research.
  • Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Nice To Haves

  • Experience and familiarity with large language models (LLMs).
  • Experience with and a solid foundation in large distributed systems, algorithms, and software engineering principles.
  • PhD in Computer Science, Machine Learning, Data Science, or a related field.
  • 5+ years of experience in a research/ML engineering or applied research scientist position, ideally focused on AI safety.
  • Hands-on experience with deep learning and transformer-based models.
  • Strong problem-solving and analytics skills, with a proactive approach to challenges.
  • Ability to thrive in fast-moving environments where priorities shift and definitions evolve.
  • Enjoy taking end-to-end ownership and learning what is necessary to get results.
  • Ability to work independently and collaborate across teams.
  • Understanding of methods for training and fine-tuning LLMs, including distillation, supervised fine-tuning, and policy optimization.

Responsibilities

  • Develop scientific frameworks for the agent lifecycle spanning quality, safety, reliability, cost, and behavioral consistency.
  • Define scientifically grounded quality signals and benchmarks for agent systems, including task success, tool-use effectiveness, plan quality, failure modes, coordination quality, and user-perceived outcomes.
  • Advance AI-system observability through approaches to trace analysis, agent health modeling, behavioral clustering, anomaly detection, and multi-agent coordination analysis.
  • Partner with engineering teams to operationalize evaluation and observability methods in production, supporting staged rollouts, experimentation, A/B testing, and automated regression detection.
  • Contribute instrumentation and semantic standards for agent observability so agent execution is more explainable, diagnosable, and comparable across systems.
  • Collaborate with product and platform teams across Foundry, Azure Monitor, and agent runtimes to shape end-to-end evaluation, benchmarking, monitoring, and investigation experiences.
  • Set scientific direction, drive research-informed product decisions, mentor others, and raise the technical bar across the organization.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service