AI Evaluation Systems Engineering Lead

Microsoft AI•Mountain View, CA
•$142,800 - $331,200

About The Position

Microsoft AI is seeking a hands-on engineering lead to build evaluation infrastructure that helps researchers improve frontier models and make reliable release decisions. The role leads the Eval Systems team in creating systems that turn model checkpoints into trustworthy results quickly, reproducibly, and at scale. The position combines technical work with team leadership. The lead shapes architecture and the roadmap, contributes to implementation, and hires and develops engineers. In partnership with research, training, and inference teams, the lead makes evaluation a dependable part of the model-development process, from early experiments through release readiness. The role calls for someone who enjoys difficult systems problems, makes thoughtful engineering tradeoffs, and builds teams grounded in clear ownership, collaboration, and technical excellence.

Requirements

  • Bachelor’s degree in Computer Science or a related technical discipline and 6+ years of technical engineering experience coding in languages including C, C++, C#, Java, JavaScript, or Python, or equivalent experience.

Nice To Haves

  • Bachelor’s degree in Computer Science or a related technical field and 10+ years of technical engineering experience coding in languages including C, C++, C#, Java, JavaScript, or Python.
  • Substantial software engineering experience building and operating production distributed systems, infrastructure platforms, or ML systems.
  • Demonstrated technical leadership, including setting direction and delivering complex projects through a team.
  • Strong programming skills, including proficiency in Python and experience contributing to large, shared codebases.
  • Experience diagnosing distributed-system failures and improving reliability, performance, and resource efficiency.
  • Experience with production operations, including monitoring, incident response, and on-call practices.
  • Ability to translate evolving requirements into clear engineering priorities and work effectively with research and engineering partners.
  • A track record of mentoring engineers and strengthening team engineering practices.
  • Experience with LLM evaluation, benchmarking, model-based grading, or reinforcement learning infrastructure.
  • Experience with GPU-backed inference, distributed training, or large-scale compute workloads.
  • Familiarity with distributed execution frameworks and schedulers such as Ray, Kubernetes, or Slurm.
  • Experience improving experimental reproducibility, validating metrics, or investigating differences across evaluation and serving environments.
  • Experience hiring engineers, managing people, or growing a technical team.
  • A track record of simplifying complex infrastructure and reducing operational burden through automation and systematic reliability improvements.

Responsibilities

  • Set technical direction and own the team’s architecture, roadmap, and delivery, balancing immediate research needs with long-term reliability and scalability.
  • Build end-to-end evaluation systems for benchmark configuration, model inference, grading, aggregation, and results publication across training checkpoints and deployed models.
  • Accelerate research iteration by reducing evaluation turnaround time and improving throughput, scheduling, and GPU efficiency; make it easier for researchers to integrate benchmarks, investigate results, and compare models.
  • Establish reproducible configurations, traceable results, validation, and regression checks, partnering with evaluation owners to preserve correctness and comparability as models and infrastructure evolve.
  • Own operational excellence by defining and tracking reliability and performance objectives and improving observability, failure recovery, and on-call practices while addressing recurring problems at their source.
  • Contribute code, lead design reviews, and help debug challenging problems across evaluation, inference, and distributed execution.
  • Hire and mentor engineers, establish clear responsibilities, and foster an inclusive environment where people can do their best work.
  • Align research and engineering partners on priorities, dependencies, and tradeoffs, and communicate progress and risks clearly.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service