L55 - Lead DevOps Engineer II

Deloitte • Nashville, TN

10d • $113,100 - $188,500 • Hybrid

About The Position

The Lead DevOps Engineer II is a hands-on production reliability and engineering role focused on the reliability, performance, cost-effectiveness, and operational integrity of high-visibility products, platforms, and their environments. The role applies cloud platform engineering, observability, performance and reliability engineering, and applied AI expertise to operate resilient systems and AI and agentic workloads at scale. The successful candidate will serve as a technical leader and mentor, collaborate across teams to establish production standards, safeguard environments, and confidently admit systems into production.

The role owns service-level objectives and error budgets, production observability, readiness verification, environment integrity, operational tooling, and production admission decisions. It includes defining and improving operational standards, conducting performance and resilience testing, automating quality checks throughout the software development lifecycle, and turning incidents into systemic improvements through blameless postmortems. The engineer partners with product and engineering teams to create lean, supportable operational solutions and continuously improve reliability, performance, and cost outcomes.

The position is part of US Deloitte Technology Product Engineering, Deloitte’s internal development team, which builds and deploys digital solutions for Deloitte’s businesses, service lines, and internal operations. Candidates must be within commuting distance of a listed location and work from a local office at least three days per week. Travel is approximately 10% on average. Due to the nature of the work, the candidate must be a U.S. citizen and legally authorized to work in the United States without current or future employer sponsorship.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, machine learning, or a related discipline; relevant experience is the most important factor.
• At least 6 years of software engineering and site reliability engineering experience operating large-scale, distributed, cloud-native systems in production.
• Experience with most of the following: Python, Go, Bash, Java, C#/.NET, SQL/NoSQL, Kubernetes, Terraform, ArgoCD, CI/CD, and observability stacks.
• At least 3 years of site reliability or production engineering experience for large-scale systems, including defining and owning SLIs, SLOs, SLAs, and error budgets; incident command and on-call; production observability; environment integrity and drift prevention; and segregation-of-duties controls.
• At least 3 years of cloud-native engineering and cloud platform ownership on Azure, AWS, or GCP, including AI/ML services, container orchestration, infrastructure-as-code, networking, and multi-environment management.
• At least 1 year establishing reliability and operational standards, including SLO discipline, runbooks, performance and resilience budgets, and leading, mentoring, and guiding teammates in adopting and improving those standards.
• Prior experience operating AI/ML and agentic workloads in production, including reliability failure modes, MLOps/LLMOps, and AI control planes such as model or LLM gateways and guardrails.
• Experience with load and performance testing under simulated production traffic, chaos engineering, capacity planning, autoscaling, and cloud/AI cost engineering, including GPU/inference and token cost attribution.
• Prior software engineering experience and understanding of Business Context Diagrams, sequence/activity/state/entity relationship/data flow diagrams, OOP/OOD, data structures, algorithms, code instrumentation, and AI-augmented spec-driven development.
• Experience with methodologies and tools such as XP, Lean, DevSecOps, SRE, ADO, GitHub, SonarQube, MLflow, and agentic AI frameworks such as LangFuse, LangSmith, or equivalent multi-agent orchestration tools.
• Ability to travel approximately 10% on average.
• Must be legally authorized to work in the United States without the need for employer sponsorship now or at any time in the future; limited immigration sponsorship may be available.
• Must be located within commuting distance of one of the listed locations and able to work in a local office at least three days per week.
• Must be a U.S. citizen due to the nature of the work.
• Excellent interpersonal and organizational skills, including the ability to handle diverse situations, complex projects, and changing priorities with passion, empathy, and care.

Responsibilities

• Own reliability, performance, and cost outcomes for supported products, platforms, and environments, using service-level objectives and error budgets to guide operational work.
• Track incident trends and operational toil to prioritize reliability improvements and maintain lean, safe, resilient production operations.
• Advocate for production reliability and operability, set production standards, and ensure systems are admissible, performant, safe to operate, and able to degrade gracefully.
• Lead the design of observability, performance and resilience testing, and operational tooling.
• Own production admission decisions, gate releases on error budgets and automated reliability checks, and verify readiness and environment integrity.
• Own SLOs and error budgets and build and operate production observability, including version-controlled dashboards and actionable, SLO-driven alerting.
• Run performance, ambient-noise, and chaos testing to verify readiness, and protect production and pre-production environments against configuration drift.
• Create technical specifications, runbooks, shared playbooks, and high-quality, supportable automation.
• Lead blameless postmortems that convert incidents into learning and systemic fixes.
• Review other engineers’ work, mentor and guide team members, and help ensure availability, performance, and cost reliability KPIs are met or exceeded.
• Partner with engineering teams before, during, and after delivery to co-define SLOs and operational readiness without creating a delivery bottleneck.
• Improve reliability incrementally through measurable changes, progressive resilience testing, and SLO refinement.
• Collaborate with engineering, platform engineering, security and risk, data governance, engineering leadership, architecture, product teams, and customers as needed.
• Enforce production controls while partnering with security and risk, including maintaining segregation-of-duties requirements.
• Operate AI and agentic workloads reliably, addressing AI-specific failure modes such as drift, train/serve skew, latency, output variance, and token or GPU cost anomalies.
• Translate product and platform domain requirements into SLOs, runbooks, and production tooling, and communicate technical recommendations and trade-offs in terms of business objectives.

Benefits

• Employees are offered a broad range of benefits.
• May be eligible to participate in a discretionary annual incentive program, subject to program rules and individual and organizational performance.
• Opportunities to build new skills, take on leadership opportunities, and connect and grow through mentorship.
• On-the-job learning experiences and formal development programs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior