Senior AI Engineer

Microsoft•Redmond, WA

14d

About The Position

Microsoft is a company where passionate innovators come to collaborate, envision what can be and take their careers further. This is a world of more possibilities, more innovation, more openness, and the sky is the limit thinking in a cloud-enabled world. Microsoft’s Azure Data engineering team is leading the transformation of analytics in the world of data with products like databases, data integration, big data analytics, messaging & real-time analytics, and business intelligence. The products our portfolio include Microsoft Fabric, Azure SQL DB, Azure Cosmos DB, Azure PostgreSQL, Azure Data Factory, Azure Synapse Analytics, Azure Service Bus, Azure Event Grid, and Power BI. Our mission is to build the data platform for the age of AI, powering a new class of data-first applications and driving a data culture. Within Azure Data, the messaging and real-time analytics team provides comprehensive solutions and a robust platform that enables users to ingest high granularity signals (real-time & observability) and complex data, converting those into a competitive advantage in real-time for both end users and modern applications. The Real-Time Intelligence team is hiring a Senior AI Engineer to build and scale the evals platform that underpins our real-time, agentic solutions on live operational data. You will focus on creating human and LLM-as-judge pipelines, and telemetry that measure how well our autonomous agents, anomaly detection, and decisioning systems perform in the real world. In this role, you’ll own end-to-end implementation of eval pipelines and tooling, integrate them into CI/CD and production workflows, and partner closely with product and platform teams so that every AI feature is measurable, debuggable, and continuously improving. Rigorous evaluation is the most critical ingredient in building reliable agents, this is a highly leveraged, key role that will directly shape how we design, ship, and operate real-time AI agentic products. We do not just value differences or different perspectives. We seek them out and invite them in so we can tap into the collective power of everyone in the company. As a result, our customers are better served

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Nice To Haves

Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
Demonstrated experience shipping LLM-powered features or agentic systems in production (e.g., copilots, assistants, workflow agents, tool-calling systems).
Experience with at least one major cloud platform (Azure, AWS, or GCP) and modern infrastructure practices (microservices, containers, orchestration, CI/CD).
Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch.
Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines.
Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders.
Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch.
Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines.
Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders.

Responsibilities

Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, and decisioning systems.
Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy to monitor and debug.
Integrate evals into the development lifecycle by connecting pipelines to CI/CD, canary and A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations.
Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features.
Embody our culture and values

Stand Out From the Crowd

Upload your resume and get instant feedback on how well it matches this job.

Upload and Match Resume