Applied AI Site Reliability Engineer III

Deloitte • Nashville, TN

7d • $102,500 - $171,000 • Hybrid

About The Position

As an Applied AI Site Reliability Engineer III, you will take a hands-on role in the reliability, performance, and operational integrity of high-visibility products, platforms, and their environments. You will help keep production safe, performant, and cost-effective by applying expertise in cloud platform engineering, observability, performance and reliability engineering, and operating AI and agentic workloads. You will collaborate across teams to uphold production standards, safeguard environments, and confidently admit systems into production.

You will own reliability outcomes measured through service-level objectives and error budgets, advocate for production readiness and operability, and build and operate observability, testing, tooling, automation, runbooks, and shared playbooks. The role includes maintaining environment integrity, leading blameless postmortems, improving reliability incrementally, and partnering with engineering, platform, security and risk, data governance, and architecture teams. For AI-enabled systems, you will address operational risks such as drift, train/serve skew, latency, output variance, and token or GPU cost anomalies.

The role is part of US Deloitte Technology Product Engineering, Deloitte’s internal development team, which delivers digital solutions for its businesses, service lines, and internal operations. The position requires working in a local office at least three days per week and being within commuting distance of a selected location. Travel averages 10%. The estimated wage range is $102,500 to $171,000, and the employee may be eligible for a discretionary annual incentive program.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; experience is identified as the most relevant factor.
• At least 5 years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native production systems, with experience in most of Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• At least 3 years of site reliability or production engineering experience for large-scale systems, including defining and owning SLIs, SLOs, SLAs, and error budgets; incident command and on-call; production observability; environment integrity and drift prevention; and segregation-of-duties controls in partnership with security and risk.
• At least 3 years of cloud-native engineering and cloud platform ownership using a hyperscaler such as Azure, AWS, or GCP, including AI/ML services, container orchestration, infrastructure as code, networking, and multi-environment management.
• Prior experience operating AI/ML and agentic workloads in production, including reliability failure modes, MLOps/LLMOps, and the AI control plane from an operability and performance perspective.
• Prior experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Prior software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Prior experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks or equivalent multi-agent orchestration tools.
• Ability to travel an average of 10%, depending on the work and products.
• Must be located within commuting distance of a selected location for the role.
• Ability to work in a local office at least 3 days per week.
• Excellent interpersonal and organizational skills, including the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.
• Strong communication, collaboration, and influencing skills; ability to articulate complex technical concepts and align technical solutions with business objectives.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, using service-level objectives and error budgets.
• Track incident trends and operational toil to prioritize improvements and maintain lean, safe, resilient production operations.
• Advocate for production reliability and operability, uphold production standards, and ensure systems are safe to run and able to degrade gracefully.
• Lead the design of observability, performance and resilience testing, and operational tooling; own production admission, readiness verification, and ongoing operational support.
• Gate releases using error budgets and automated reliability checks, while maintaining environment integrity and production and pre-production standards.
• Define and own SLOs and error budgets; build and operate version-controlled dashboards and actionable, SLO-driven alerts that detect issues before impact.
• Run performance, ambient-noise, and chaos testing; prevent environment drift; and create technical specifications, runbooks, and shared playbooks.
• Write and maintain high-quality automation to meet availability, performance, and cost reliability KPIs.
• Lead blameless postmortems and turn incidents into learning and systemic improvements.
• Develop lean operational solutions through rapid experimentation and collaborate with engineering teams to define SLOs and operational readiness without becoming a delivery bottleneck.
• Improve reliability incrementally through measurable resilience testing and SLO refinement, keeping operations supportable and maintainable.
• Collaborate with engineering, platform engineering, security and risk, data governance, engineering leadership, and architecture; integrate relevant constraints and uphold segregation-of-duties controls.
• Operate AI and agentic workloads reliably, including their production operations and quality checks across the AI and Agentic SSDLC lifecycle.
• Learn the products and platforms being operated and translate reliability needs, reference architectures, and operational requirements into SLOs, runbooks, and production tooling.
• Communicate technical concepts, trade-offs, and evidence clearly, align solutions with business objectives, and build constructive relationships with product engineering teams and customers as needed.

Benefits

• Broad range of employee benefits (specific benefits are not listed in the posting).
• May be eligible to participate in a discretionary annual incentive program, subject to program rules and applicable factors.
• Opportunities for professional development, including on-the-job learning, leadership opportunities, mentorship, and formal development programs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior