Lead Applied AI Site Reliability Engineer II

Deloitte • Tampa, FL

23h • $113,100 - $208,300 • Hybrid

About The Position

The Lead Applied AI Site Reliability Engineer II takes a hands-on role in the reliability, performance, and operational integrity of high-visibility products, platforms, and their environments. The role keeps production safe, performant, and cost-effective by applying cloud platform engineering, observability, performance and reliability engineering, and applied AI expertise to operate AI and agentic workloads alongside other systems. The successful candidate is a role-model leader and mentor who collaborates across teams to establish production standards, safeguard environments, and confidently admit systems into production.

The position emphasizes outcome-based reliability through service-level objectives and error budgets; production readiness and admission standards; operational integrity across production and pre-production environments; and continuous, measurable improvement. The engineer owns observability, performance and resilience testing, environment integrity, incident learning, automation, runbooks, and operational tooling, while partnering with product engineering, platform engineering, security and risk, data governance, and engineering leadership. The role operates AI-infused systems with attention to failure modes such as drift, train/serve skew, latency, output variance, and token or GPU cost anomalies.

US Deloitte Technology Product Engineering develops and deploys internal and go-to-market digital solutions for Deloitte businesses, service lines, and operations. The position is available in Dallas, Texas; Hermitage, Tennessee; Nashville, Tennessee; and Tampa, Florida. Candidates must be within commuting distance of a listed location, work from their local office at least three days per week, and travel an average of 10%. Limited immigration sponsorship may be available.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; experience is identified as the most relevant factor.
• At least 6 years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native production systems.
• Experience with most of the following: Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• At least 3 years of site reliability or production engineering experience defining and owning SLIs, SLOs, SLAs, error budgets, incident command, and on-call practices.
• At least 3 years of experience building and operating production observability, including metrics, tracing, and logging; examples include OpenTelemetry, Prometheus, Grafana, Datadog, Dynatrace, Amazon CloudWatch, Azure Monitor, Google Cloud Operations, SolarWinds, and Splunk.
• Experience maintaining environment integrity and preventing drift across pre-production and production environments.
• Experience with segregation-of-duties controls, including least privilege/RBAC, deployment approvals, and secrets management, in partnership with security and risk.
• At least 3 years of cloud-native engineering and cloud platform ownership on Azure, AWS, or GCP, including relevant AI/ML services such as Azure OpenAI, AWS Bedrock, or Vertex AI; container orchestration, infrastructure-as-code, networking, and multi-environment management.
• At least 1 year of experience establishing reliability and operational standards, including SLO discipline, runbooks, and performance and resilience budgets, and actively leading, mentoring, and guiding team members in adopting and improving those standards.
• Prior experience operating AI/ML and agentic workloads in production, including reliability failure modes such as drift, train/serve skew, and output variance; MLOps/LLMOps; and the AI control plane, such as model/LLM gateways and guardrails, from an operability and performance perspective.
• Experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Prior software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data-flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks such as LangFuse, LangSmith, or equivalent multi-agent orchestration tools.
• Excellent interpersonal and organizational skills, including the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.
• Must be located within commuting distance of one of the role’s listed locations.
• Must be able to work in the local office at least three days per week.
• Must be able to travel an average of 10%, based on the work and products supported.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, measuring results through SLOs and error budgets.
• Track incident trends and operational toil to prioritize improvements that strengthen reliability and keep production safe and resilient.
• Advocate for production reliability and operability, establish production standards, and ensure systems are safe, performant, and able to degrade gracefully.
• Lead observability, performance, resilience-testing, and operational-tooling design; own production admission decisions and gate releases using error budgets and automated reliability checks.
• Verify operational readiness and maintain the integrity of production and pre-production environments, including preventing environment drift.
• Define and operate SLOs, error budgets, version-controlled dashboards, actionable alerting, and engineering feedback loops.
• Conduct performance, ambient-noise, and chaos testing to verify production readiness.
• Create technical specifications, runbooks, shared playbooks, and high-quality, supportable automation.
• Lead blameless postmortems and translate incidents into learning and systemic fixes.
• Review engineers’ work, mentor and guide team members, and help meet or exceed availability, performance, and cost KPIs.
• Develop lean operational solutions through rapid, inexpensive experimentation, engaging engineering teams before, during, and after delivery.
• Improve reliability incrementally through measurable resilience testing and SLO refinement while keeping operations maintainable and supportable.
• Collaborate with engineering, platform engineering, security and risk, data governance, architecture, and leadership to integrate constraints into production standards and controls.
• Partner with security and risk on control objectives while maintaining segregation of duties as an embedded function.
• Operate AI and agentic workloads reliably, applying AI and Agentic SSDLC practices and automation across discovery, production, operations, and quality checks.
• Acquire domain knowledge and translate reliability needs, reference architectures, and operational requirements into SLOs, runbooks, and production tooling.
• Communicate technical concepts, evidence-based trade-offs, and business-aligned solutions; build relationships and consensus with product teams and customers as needed.

Benefits

• Broad range of employee benefits.
• May be eligible to participate in a discretionary annual incentive program, subject to program rules and applicable performance factors.
• Professional development opportunities, including on-the-job learning, formal development programs, leadership opportunities, and mentorship.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior