Applied AI Site Reliability Engineer III

Deloitte • Nashville, TN

23h • $102,500 - $188,900 • Hybrid

About The Position

As an Applied AI Site Reliability Engineer III, you will take a hands-on role in the reliability, performance, and operational integrity of high-visibility products, platforms, and their environments. You will help keep production safe, performant, and cost-effective by applying cloud platform engineering, observability, performance and reliability engineering, and applied AI expertise to operate AI and agentic workloads alongside other systems. You will collaborate with cross-functional teams to uphold production standards, safeguard environments, and confidently admit systems into production.

You will own reliability outcomes measured through service-level objectives (SLOs) and error budgets, rather than uptime alone. This includes building and operating observability, testing performance and resilience, preventing environment drift, creating automation and operational documentation, leading blameless postmortems, and improving reliability, performance, and cost. You will partner with engineering, platform, security and risk, data governance, architecture, and product teams to define objectives and readiness criteria, verify production readiness, and make production admission decisions without becoming a delivery bottleneck.

The role requires modern site reliability and production engineering expertise, cloud platform ownership, and the ability to operate AI and agentic workloads reliably throughout their lifecycle. The successful candidate will acquire domain knowledge of supported products, translate reliability and operational needs into SLOs and tooling, communicate technical trade-offs effectively, and contribute to incremental, measurable improvements.

The role is part of US Deloitte Technology Product Engineering, Deloitte’s primary internal development team. The team delivers digital solutions for Deloitte businesses, service lines, and internal operations, as well as internal and go-to-market products. The role is available in Dallas, Texas; Hermitage, Tennessee; or Nashville, Tennessee. Candidates must live within commuting distance of the selected location and work from their local office at least three days per week. Travel averages 10%.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; the posting states that experience is the most relevant factor.
• 5+ years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native systems in production.
• Experience with most of the following: Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• 3+ years of site reliability or production engineering experience for large-scale systems, including defining and owning SLIs, SLOs, SLAs, and error budgets; incident command and on-call; production observability; environment integrity and drift prevention; and segregation-of-duties controls in partnership with security and risk.
• 3+ years of cloud-native engineering and cloud platform ownership on Azure, AWS, or GCP, including AI/ML services, container orchestration, infrastructure as code, networking, and multi-environment management.
• Experience operating AI/ML and agentic workloads in production, including reliability failure modes such as drift, train/serve skew, and output variance; MLOps/LLMOps; and AI control-plane components such as model or LLM gateways and guardrails.
• Experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks such as LangFuse, LangSmith, or equivalent multi-agent orchestration tools.
• Ability to travel an average of 10%, based on the work and products built.
• Must be located within commuting distance of the selected role location.
• Must be able to work in the local office at least three days per week.
• Excellent interpersonal and organizational skills, with the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.
• Strong communication, collaboration, and relationship-building skills.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, using SLOs and error budgets.
• Monitor incident trends and operational toil, and prioritize improvements that increase reliability and reduce operational overhead.
• Advocate for production reliability and operability; uphold production standards and ensure systems are safe, performant, resilient, and able to degrade gracefully.
• Lead the design of observability, performance and resilience testing, and operational tooling.
• Own production admission decisions, gating releases on error budgets and automated reliability checks, and perform readiness verification and operational support.
• Maintain the operational integrity of production and pre-production environments and prevent environment drift.
• Build and operate codified, version-controlled dashboards and SLO-driven actionable alerting, and feed operational findings back into engineering.
• Run performance, ambient-noise, and chaos testing to verify readiness.
• Create technical specifications, runbooks, and shared playbooks; lead blameless postmortems and turn incidents into systemic fixes and learning.
• Develop and maintain high-quality automation to meet availability, performance, and cost KPIs.
• Collaborate with engineering and product teams before, during, and after delivery to co-define SLOs and operational readiness requirements.
• Deliver incremental, measurable reliability improvements through resilience testing and SLO refinement.
• Partner with engineering, platform engineering, security and risk, data governance, engineering leadership, and architecture; integrate relevant constraints into production standards and controls.
• Operate AI and agentic workloads reliably, including their distinct failure modes and production controls.
• Acquire domain knowledge of supported products and platforms, and translate reliability needs, reference architectures, and operational requirements into SLOs, runbooks, and production tooling.
• Communicate technical concepts, evidence-based trade-offs, and solutions in alignment with business objectives; build constructive relationships and consensus across teams.

Benefits

• Eligible to participate in a discretionary annual incentive program, subject to program rules and applicable factors.
• Broad range of employee benefits.
• Professional development opportunities, including on-the-job learning, leadership opportunities, mentorship, and formal development programs.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level