Senior Software Engineer

Microsoft•Redmond, WA

2d•$119,800 - $261,000

About The Position

How do you know an AI product is actually getting better, and how do you prove it, at scale, before millions of people feel the difference? We build and operate the offline evaluation platform that gates Microsoft Copilot’s quality: teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. We do this at the scale of one of the world’s largest AI products, inside a strict enterprise compliance boundary. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will be one of the engineers building it. As a Senior Software Engineer on the Copilot evaluation team, you will be a key engineer on a platform designed to be AI-first and agentic-first, where autonomous agents are not just measured, but help operate and extend the system itself. You will design and deliver major components of the platform and partner closely with senior and principal engineers on the systems that keep evaluation trustworthy across a deep chain of dependencies we drive far outside their normal operating envelope, hundreds of calls per job, under relentless and growing load. You will build the capabilities that let us evaluate agentic Copilot experiences, from multi-turn trajectories and tool selection to end-to-end task outcomes. This opportunity will allow you to deepen your expertise in large-scale agentic systems and reliability engineering, broaden your impact across one of the world’s most visible AI products, and grow toward technical leadership on problems the industry has not solved yet. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python OR equivalent experience.
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role.
Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Nice To Haves

Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python OR equivalent experience.
Experience with large-scale distributed systems and reliability engineering, especially building dependable services that operate across many interacting dependencies under heavy or unusual load.
Experience building AI-first or agentic systems, including systems, APIs, or tooling intended to be operated by autonomous agents, with the observability and guardrails that make agent-driven operation trustworthy.
Experience with AI/ML evaluation, measurement, or benchmarking systems, such as LLM-based grading, metric design, or large-scale data and quality pipelines.

Responsibilities

Partners with engineers and stakeholders across the Copilot organization to turn quality questions into rigorous, reproducible evaluations, building the scenarios, metrics, and scorecards that gate what ships.
Designs and builds major components of an AI-first, agentic-first evaluation platform where autonomous agents are first-class operators, delivering services, pipelines, and tooling that agents can run, extend, and reason about, with the observability and guardrails that make agent-driven operation trustworthy.
Contributes to the platform’s hardest challenge, delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope, implementing graceful degradation, intelligent retry, and dependency-aware gating so reliability holds under sustained, growing load.
Builds the capabilities that make agentic Copilot experiences measurable end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, including the simulation, scraping, and scoring that capture agent behavior.
Writes and reviews extensible, maintainable code and mentors less-experienced engineers, contributing to modernization of the evaluation runtime and platform so it scales with fast-growing demand while improving cost and latency.
Acts as a Designated Responsible Individual (DRI) for a system that gates releases, helping instrument the reliability, throughput, and dependency health of the platform and working on-call so the results teams depend on stay correct and timely.
Stays current with agentic systems, large-scale AI evaluation, and autonomous operations, adopting new tools and patterns and sharing what works to raise the bar for how the team measures quality.