Lead Associate Principal, Software Engineering: DevOps

OCC•Chicago, IL
•$140,800 - $232,500•Hybrid

About The Position

The Lead Associate Principal, Software Engineering: DevOps role leads and optimizes L1 and L2 Environment Operations support for production application deployments, middleware, and platform infrastructure. The role is responsible for incident-response quality, SLA governance, alert reduction, MTTR improvement, and delivering operational metrics to leadership. It ensures environment issues are triaged, resolved, or escalated promptly and accurately, with complete documentation. The position leads incident response, serves as a Tier 3 escalation point for complex issues, owns incidents through resolution and root-cause documentation, and drives post-incident reviews for P1 and P2 incidents. It establishes and monitors severity-based SLA targets, reports compliance and trends, reduces alert noise, improves operational metrics, and ensures recurring issues lead to permanent fixes. The role also owns the runbook library and maintains current environment configuration documentation and operational procedures. The position requires at least seven years of hands-on environment operations, production support, or infrastructure operations experience, including work across four or more relevant disciplines. Previous people management or team lead experience is required, with formal people management experience strongly preferred. Shift work and on-call availability are required, including 24×7 on-call response capacity and availability during planned and emergency maintenance windows. The position is hybrid, with up to two remote-work days per week.

Requirements

  • At least seven years of hands-on environment operations, production support, or infrastructure operations experience.
  • Interdisciplinary experience across four or more of the following: application deployment pipelines, container platform operations, middleware support, incident management, monitoring and observability, configuration management, release engineering, platform operations, or scripting and automation.
  • Technical experience and comprehensive knowledge of production environment failure modes, including deployment failures, configuration drift, platform instability, and integration breakdowns, and methods to diagnose and resolve them.
  • Demonstrated experience defining and enforcing SLA frameworks in a tiered support model such as L1/L2/L3 or equivalent.
  • Previous people management or team lead experience; formal people management experience is strongly preferred.
  • Proven team leadership, initiative, follow-through, and ability to hold a team accountable to defined standards across incident types and urgencies.
  • Ability to work effectively across L1, L2, Platform, Security, and App Dev teams on improvement programs and rapid incident response under tight deadlines.
  • Deep hands-on production environment support experience, including application deployment pipelines, container orchestration, messaging platforms, and middleware infrastructure.
  • Ability to independently create, tune, and maintain monitoring alerts and operational runbooks.
  • Excellent oral and written communication skills, including translating technical incident details into accurate, concise leadership reporting.
  • Strong analytical, judgment, and consultation skills; ability to triage ambiguity, make sound decisions under pressure, and consult with technical and non-technical stakeholders.
  • Ability to work independently and manage multiple parallel priorities with strong organizational discipline.
  • Experience with Harness continuous delivery pipelines, deployment verification, and rollback automation; Jenkins CI/CD pipeline management, job configuration, and build troubleshooting; and GitHub branching strategies, pull request workflows, and pipeline integration.
  • Kubernetes pod lifecycle management, namespace operations, log retrieval, resource troubleshooting, and coordination with Platform teams on cluster-level issues.
  • Apache Kafka topic management, consumer group monitoring, lag analysis, and escalation of broker-level issues.
  • HashiCorp Vault secrets retrieval, token and lease troubleshooting, policy review, and escalation to Security teams for certificate and secrets rotation.
  • Proficiency in at least two production monitoring toolsets, such as Splunk, Dynatrace, Datadog, AppDynamics, or Prometheus/Grafana, including alert triage, dashboard interpretation, log analysis, and tuning requests.
  • Working knowledge of middleware infrastructure, including application servers, messaging brokers, storage integrations, and network-layer dependencies, sufficient to triage, gather diagnostics, and route issues to L3.
  • ServiceNow or equivalent ITSM tooling experience for incident creation, SLA tracking, problem record management, and reporting.
  • Shift work and on-call availability, including 24×7 on-call response capacity and availability during planned and emergency maintenance windows.
  • Certificates or licenses are not required.

Nice To Haves

  • Familiarity with financial services or other regulated-industry production environments, including change governance, audit requirements, and production access controls.
  • Certified Kubernetes Administrator (CKA) or equivalent platform certification.
  • ITIL Foundation or above.
  • HashiCorp Vault Associate certification.
  • Harness Certified Continuous Delivery Architect certification.
  • Relevant associate-level or higher cloud platform certification in AWS, Azure, or GCP.
  • Formal people management experience.

Responsibilities

  • Lead L1 and L2 support engineers in incident triage, investigation, coordination, resolution, closure, and post-incident reporting.
  • Oversee technical analysis of incidents across application deployments, middleware, and platform layers, coordinating with engineering, platform, and application development teams.
  • Serve as the Tier 3 escalation point for complex incidents involving Platform, S&I, Security, and App Dev teams.
  • Own the full incident lifecycle, from first alert through root-cause analysis documentation and permanent fix or accepted workaround.
  • Lead post-incident reviews for all P1 and P2 incidents and ensure identified root causes are documented and actioned.
  • Define, publish, and enforce SLA targets across severity levels; monitor compliance, escalate breaches, and report trends to leadership.
  • Ensure incident priority classifications are accurate, consistent, and applied during triage.
  • Publish monthly SLA compliance reports with trend analysis and improvement actions.
  • Lead alert tuning and noise-reduction initiatives across monitoring toolsets.
  • Track and publish alert-to-incident ratios each sprint and drive an improving trend.
  • Improve and report MTTR by environment, severity, and team tier, including quarterly trend commentary and actions.
  • Identify recurring incident patterns, open Problem records, and drive permanent fixes rather than repeated workarounds.
  • Lead automation and tooling initiatives to reduce toil, accelerate triage, and eliminate manual support-workflow steps.
  • Ensure every incident is documented with symptoms, actions, diagnostics, resolution, and root-cause analysis where applicable.
  • Own the runbook library and ensure each novel resolution is documented in a runbook published to L1 before incident closure; track and close coverage gaps.
  • Maintain current, accessible, and team-reviewed environment configuration documentation and operational procedures.
  • Build and maintain operational dashboards and reports for MTTR, SLA compliance, alert-to-incident ratio, repeat incident rate, and deployment success rate.

Benefits

  • Hybrid work environment with up to two days per week of remote work.
  • Tuition reimbursement.
  • Student loan repayment assistance.
  • Technology stipend for a device of the employee's choice to connect to the network while working remotely.
  • Generous paid time off and parental leave.
  • 401(k) employer match.
  • Competitive medical, dental, and vision benefits.
  • Collaborative and supportive environment intended to encourage work-life balance and employee wellness.
  • Employees may be eligible for an annual discretionary incentive compensation award; the target incentive range is 8% to 15%, and awards are not guaranteed.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service