Applied AI SRE III - PxE GPS

Deloitte • Nashville, TN

2d • $102,500 - $188,900 • Hybrid

About The Position

The Applied AI Site Reliability Engineer III takes a hands-on role in the reliability, performance, and operational integrity of high-visibility products, platforms, and their environments. The engineer helps keep production safe, performant, and cost-effective, applying cloud platform engineering, observability, performance and reliability engineering, and applied AI expertise to operate AI and agentic workloads alongside other systems. The role collaborates across teams to uphold production standards, safeguard environments, and confidently admit systems into production.

The position owns reliability and operational outcomes measured through service-level objectives and error budgets, not raw uptime. It establishes production standards, designs observability and resilience testing, owns readiness verification and production admission, and maintains production and pre-production environment integrity. Responsibilities include building actionable, version-controlled observability and automation; running performance, ambient-noise, and chaos tests; preventing environment drift; authoring specifications, runbooks, and playbooks; and leading blameless postmortems that drive systemic improvement.

The engineer works closely with product and engineering teams to define SLOs and operational readiness, delivering lean, incremental solutions without becoming a delivery bottleneck. The role partners with platform engineering, security and risk, data governance, and engineering leadership and architecture, integrating applicable constraints and controls. For AI-infused systems, the engineer learns and addresses production failure modes such as drift, train/serve skew, latency, output variance, and token or GPU cost anomalies, translating reliability needs into SLOs, runbooks, and production tooling.

The position is part of US Deloitte Technology Product Engineering, Deloitte’s internal development team, which delivers digital solutions for businesses, service lines, and internal operations. The role is based within commuting distance of one of the listed locations, requires work in the local office at least three days per week, and involves an average of 10% travel. The candidate may also be eligible for a discretionary annual incentive program.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; experience is identified as the most relevant factor.
• At least 5 years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native production systems.
• Experience with most of the following: Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• At least 3 years of site reliability or production engineering experience for large-scale systems, including defining and owning SLIs, SLOs, SLAs, and error budgets; incident command and on-call; production observability; environment integrity and drift prevention; and segregation-of-duties controls in partnership with security and risk.
• At least 3 years of cloud-native engineering and cloud platform ownership on Azure, AWS, or GCP, including related AI/ML services, container orchestration, infrastructure as code, networking, and multi-environment management.
• Prior experience operating AI/ML and agentic workloads in production, including reliability failure modes, MLOps/LLMOps, and the AI control plane from the operability and performance perspective.
• Prior experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Prior software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Prior experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks such as LangFuse, LangSmith, or equivalent multi-agent orchestration tools.
• Excellent interpersonal and organizational skills, including the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.
• Must be legally authorized to work in the United States without employer sponsorship now or in the future.
• Must be located within commuting distance of one of the listed locations and able to work in the local office at least three days per week.
• Ability to travel an average of 10%, based on the work performed and products built.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, using SLOs and error budgets to guide operational work.
• Track incident trends and operational toil to prioritize improvements that make systems more reliable and production operations leaner.
• Advocate for production reliability and operability, uphold production standards, and ensure systems are safe to run and can degrade gracefully.
• Lead the design of observability, performance and resilience testing, and operational tooling.
• Own system admission into production, including readiness verification, environment integrity, release gating against error budgets and automated reliability checks, and operational support.
• Define and own SLOs and error budgets, and build and operate codified, version-controlled dashboards and actionable SLO-driven alerting.
• Run performance, ambient-noise, and chaos testing to verify readiness, and prevent environment drift across production and pre-production.
• Create technical specifications, runbooks, and shared playbooks; lead blameless postmortems that produce learning and systemic fixes.
• Write supportable automation to meet or exceed availability, performance, and cost reliability KPIs.
• Develop lean operational solutions through rapid experimentation and collaborate with engineering teams before, during, and after delivery to define SLOs and operational readiness.
• Improve reliability incrementally through measurable resilience testing and SLO refinement, keeping operations supportable and maintainable.
• Collaborate with engineering, platform engineering, security and risk, data governance, and engineering leadership and architecture; integrate their constraints into production practices.
• Enforce production controls and segregation of duties while partnering with security and risk on control objectives.
• Operate AI and agentic workloads reliably, including applying AI and Agentic SSDLC practices and automation across the lifecycle from discovery through production operations and quality checks.
• Acquire domain knowledge of supported products and platforms and translate reliability needs, reference architectures, and operational requirements into SLOs, runbooks, and production tooling.
• Communicate technical concepts and trade-offs clearly, influence teams with evidence, and build collaborative relationships with product engineering teams and customers as needed.

Benefits

• Potential eligibility for a discretionary annual incentive program, subject to program rules and applicable individual and organizational performance factors.
• A broad range of employee benefits.
• Professional development opportunities, including on-the-job learning, leadership opportunities, mentorship, and formal development programs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior