Lead Applied AI Site Reliability Engineer II - PxE ERM

Deloitte • Nashville, TN

2d • $113,100 - $208,300 • Hybrid

About The Position

The Lead Applied AI Site Reliability Engineer II takes a hands-on role in the reliability, performance, and operational integrity of high-visibility products, platforms, and their environments. The role keeps production safe, performant, and cost-effective by applying cloud platform engineering, observability, performance and reliability engineering, and applied AI expertise to operate AI and agentic workloads alongside other systems. The successful candidate is a technical leader and mentor who collaborates across teams to set production standards, safeguard environments, and admit systems into production with confidence.

The engineer owns service-level objectives (SLOs) and error budgets; builds and operates production observability; leads performance, resilience, and chaos testing; protects environments against configuration drift; and drives readiness verification and production admission. The role also includes incident response and blameless postmortems, technical specifications and runbooks, reliable automation, engineering reviews, and mentoring. The engineer partners with product, platform, security, risk, data governance, and architecture teams to define operational readiness and enforce production controls without unnecessarily slowing delivery.

Deloitte Product Engineering develops and deploys internal and go-to-market digital solutions. The role is available in Dallas, Texas; Hermitage, Tennessee; and Nashville, Tennessee. Candidates must be within commuting distance of a listed location and work from their local office at least three days per week. Travel averages 10%. The stated wage range is $113,100 to $208,300, and the employee may also be eligible for a discretionary annual incentive, subject to program rules and applicable factors.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; experience is described as the most relevant factor.
• At least 6 years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native production systems, with experience in most of Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• At least 3 years of site reliability or production engineering experience defining and owning SLIs, SLOs, SLAs, and error budgets; incident command and on-call; production observability; environment integrity and drift prevention; and segregation-of-duties controls in partnership with security and risk.
• At least 3 years of cloud-native engineering and cloud platform ownership on Azure, AWS, or GCP, including applicable AI/ML services, container orchestration, infrastructure as code, networking, and multi-environment management.
• At least 1 year establishing reliability and operational standards such as SLO discipline, runbooks, and performance and resilience budgets, while leading, mentoring, and guiding team members in adopting and improving those standards.
• Prior experience operating AI/ML and agentic workloads in production, including reliability failure modes, MLOps/LLMOps, and the operability and performance aspects of the AI control plane, model/LLM gateways, and guardrails.
• Prior experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Prior software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Prior experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks such as LangFuse, LangSmith, or equivalent multi-agent orchestration tools.
• Ability to travel an average of 10%, depending on the work and products.
• Must be legally authorized to work in the United States without employer sponsorship now or at any time in the future; limited immigration sponsorship may be available.
• Must be located within commuting distance of one of the listed locations and able to work in the local office at least three days per week.
• Excellent interpersonal and organizational skills, including the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.
• Strong collaboration and communication skills, including the ability to explain complex technical concepts, influence teams with evidence-based arguments, and align technical solutions with business objectives.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, using SLOs and error budgets to guide priorities.
• Track incident trends and operational toil and prioritize improvements that strengthen reliability and maintain lean operational designs.
• Advocate for production reliability and operability; establish production standards and ensure systems are safe, performant, and able to degrade gracefully.
• Lead observability, performance, resilience-testing, and operational-tooling design, and own system admission into production using error budgets and automated reliability checks.
• Verify operational readiness, protect environment integrity, and provide operational support across production and pre-production environments.
• Build and operate codified, version-controlled dashboards and actionable, SLO-driven alerting, and establish feedback loops into engineering.
• Run performance, ambient-noise, and chaos testing; guard environments against drift; and refine reliability through incremental, measurable improvements.
• Create technical specifications, runbooks, and shared playbooks; write high-quality automation; and lead blameless postmortems that produce systemic fixes.
• Review engineers’ work, mentor and guide team members, and promote continuous improvement of reliability standards and KPIs.
• Collaborate with product and engineering teams before, during, and after delivery to define SLOs and operational readiness without becoming a delivery bottleneck.
• Partner with engineering, platform, security and risk, data governance, leadership, and architecture teams to integrate constraints and enforce production controls, including segregation of duties.
• Operate AI and agentic workloads reliably, accounting for failure modes such as drift, train/serve skew, latency, output variance, and token or GPU cost anomalies.
• Translate domain and operational requirements into SLOs, runbooks, and production tooling, and communicate technical trade-offs in terms aligned with business objectives.
• Engage with teams and customers as needed to build constructive relationships, align perspectives, and co-create feasible solutions.
• Travel an average of 10% and work in a local office at least three days per week.

Benefits

• Potential eligibility for a discretionary annual incentive program, subject to program rules and applicable factors.
• Deloitte offers employees a broad range of benefits; specific benefit offerings are not detailed in the posting.
• Opportunities for professional development, including on-the-job learning, formal development programs, leadership opportunities, mentorship, and skill building.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior