Principal Site Reliability Engineer

Tandem Diabetes Care

4d•$165,000 - $185,000•Remote

About The Position

The Principal Site Reliability Engineer (SRE) is responsible for the reliability, availability, and performance of the company's production systems. This role leads day-to-day production support and incident response and progressively replaces reactive firefighting with engineered SRE practice: SLOs, observability, on-call design, runbooks, and automation. It also advances infrastructure automation (Terraform/IaC), CI/CD reliability, disaster recovery, and security and compliance readiness in partnership with cross-functional teams. The role works with a distributed team that includes offshore consulting partners and is expected to raise their capability and independence; success is measured as much by what the team can do without the Principal SRE as by what the Principal SRE delivers personally. The Principal Site Reliability Engineer (SRE)'s at Tandem are also responsible for: Production Support & Incident Management Leads day-to-day production support: intake, triage, prioritization, escalation, queue health, and change execution. Establishes consistent support practices across a distributed team, including shift handoffs, ticket quality standards, and clear ownership of open issues. Leads incident management end-to-end: incident command, stakeholder communication, and blameless postmortems with corrective actions tracked to closure. Participates in and coordinates response activities for production security incidents, partnering with Security teams to contain threats, restore services, validate controls, and implement corrective actions. Owns on-call strategy: rotation design, escalation paths, alert tuning, and tooling (e.g., PagerDuty, New Relic), with explicit goals of reducing alert fatigue and building coverage that works across time zones. Participates as a senior escalation tier for high-severity incidents. Builds runbooks that standardize response to common failure modes and enable first-line resolution by engineers who did not build the system. Reliability Engineering & Observability Defines and owns SLIs and SLOs for critical services, using them to guide monitoring, alerting, and reliability priorities. Drives systemic reduction of MTTD and MTTR through better instrumentation, alerting, diagnostics, and automation. Converts recurring support burden into permanent fixes, automation, or documentation rather than absorbing it as ongoing manual work. Establishes and maintains technology currency and lifecycle management practices for production platforms, ensuring cloud services, Kubernetes clusters, operating systems, runtimes, and infrastructure components remain supported, secure, and aligned with organizational standards. Proactively identifies and mitigates End-of-Life (EOL), End-of-Support (EOS), and technology obsolescence risks. Owns business continuity and disaster recovery readiness for production platforms, including backup and recovery strategies, recovery testing, failover capabilities, recovery runbooks, and adherence to defined RTO/RPO objectives. Infrastructure Automation & CI/CD Leads infrastructure automation with Terraform, ensuring infrastructure is version-controlled, modular, reusable, and auditable, working within and improving existing patterns where they are sound. Eliminates toil through automation, reducing manual, repetitive operational work across the team. Adds reliability guardrails to CI/CD pipelines (automated rollback, change-risk checks, progressive delivery) with the teams that own them. Operational Governance and Compliance Maintains production systems in accordance with applicable regulatory and compliance requirements, ensuring audit readiness and the ongoing effectiveness of access management, change management, logging, vulnerability remediation, patch management, and other operational controls. Mains documentation and evidence supporting business continuity and disaster recovery controls, including recovery testing results, remediation plans, and audit artifacts. Partners with Security, Quality, and Compliance teams to maintain regulatory compliance, support internal and external audits, and drive timely resolution of audit findings and remediation activities. Leadership, Mentorship & Collaboration Actively grows the capability of SRE and DevOps engineers, including consulting-partner engineers, through pairing, design and code review, and incident debriefs. Creates conditions in which junior and contract engineers raise concerns and disagree openly, recognizing that silent agreement is a production risk. Communicates in a documentation-first manner suited to a distributed, multi-time-zone team, so context is durable rather than held by one person. Partners with software engineering, QA, and architecture to embed reliability and operability into the development lifecycle rather than only after incidents. Contributing Responsibilities (partners with other owners): Informs capacity planning and scaling strategy with the Test team (who own load testing and modeling) and introduces proactive resilience testing such as game days once baseline support and observability are stable. Partners with application, architecture, and business stakeholders to align business continuity and disaster recovery capabilities with application requirements and recovery objectives. Supports cloud cost optimization: rightsizing, reserved capacity, and observability spend governance. Ensures work complies with company policies, including Privacy/HIPAA and other regulatory, legal, and safety requirements. Other duties as assigned.

Requirements

Forward-thinking professional who actively leverages AI to enhance, automate, and optimize internal tooling and system integrations.
Demonstrated experience leading production support and incident management for production systems, including incident command during high-severity events.
Strong grounding in SRE principles: SLIs/SLOs, blameless postmortems, toil reduction, and treating reliability as an engineering discipline rather than a purely operational function.
Demonstrated experience owning on-call strategy, including rotation design, alert tuning, and escalation.
Expertise with Terraform or comparable IaC at scale: module design, state management, and policy-as-code guardrails.
Hands-on experience building CI/CD pipelines with reliability guardrails using tools such as GitHub Actions, Octopus Deploy, or Azure DevOps.
Deep experience with at least one major cloud platform (AWS, Azure, or GCP; [ preferred]) and with containerization and orchestration (Docker/Kubernetes).
Working knowledge of observability tooling (e.g., Prometheus, Grafana, Datadog, CloudWatch, ELK/OpenSearch) and using it to drive SLO-based alerting.
Experience designing and testing disaster recovery, including backup/restore, failover, and RTO/RPO validation.
Working knowledge of cloud security and compliance practices (IAM, network segmentation, encryption, vulnerability and patch management) and cloud cost optimization.
Proficiency in at least one scripting or programming language (e.g., Python, Go, Bash) for automation and tooling.
Experience in FDA and ISO regulated industries and with agile methodologies preferred.
10+ years in Site Reliability Engineering, DevOps, or infrastructure engineering, including:
Leading production support, incident management, and postmortem practice.
Managing production cloud infrastructure as code (Terraform or equivalent) at scale.
Operating and improving CI/CD pipelines in a production environment.
Designing, executing, and testing disaster recovery plans.
Partnering with security teams on production security and compliance posture.
2+ years mentoring or technically leading other engineers, including engineers who are remote, offshore, or contracted through a partner organization.

Nice To Haves

B.S. in Computer Science or an equivalent combination of education and applicable job experience, including technical school training and certifications in networks, servers, and cloud infrastructure; demonstrated production experience weighs more heavily than degree.
Relevant cloud certifications (e.g., AWS/Azure/GCP Professional or Architect level) preferred.

Responsibilities

Leads day-to-day production support: intake, triage, prioritization, escalation, queue health, and change execution.
Establishes consistent support practices across a distributed team, including shift handoffs, ticket quality standards, and clear ownership of open issues.
Leads incident management end-to-end: incident command, stakeholder communication, and blameless postmortems with corrective actions tracked to closure.
Participates in and coordinates response activities for production security incidents, partnering with Security teams to contain threats, restore services, validate controls, and implement corrective actions.
Owns on-call strategy: rotation design, escalation paths, alert tuning, and tooling (e.g., PagerDuty, New Relic), with explicit goals of reducing alert fatigue and building coverage that works across time zones. Participates as a senior escalation tier for high-severity incidents.
Builds runbooks that standardize response to common failure modes and enable first-line resolution by engineers who did not build the system.
Defines and owns SLIs and SLOs for critical services, using them to guide monitoring, alerting, and reliability priorities.
Drives systemic reduction of MTTD and MTTR through better instrumentation, alerting, diagnostics, and automation.
Converts recurring support burden into permanent fixes, automation, or documentation rather than absorbing it as ongoing manual work.
Establishes and maintains technology currency and lifecycle management practices for production platforms, ensuring cloud services, Kubernetes clusters, operating systems, runtimes, and infrastructure components remain supported, secure, and aligned with organizational standards. Proactively identifies and mitigates End-of-Life (EOL), End-of-Support (EOS), and technology obsolescence risks.
Owns business continuity and disaster recovery readiness for production platforms, including backup and recovery strategies, recovery testing, failover capabilities, recovery runbooks, and adherence to defined RTO/RPO objectives.
Leads infrastructure automation with Terraform, ensuring infrastructure is version-controlled, modular, reusable, and auditable, working within and improving existing patterns where they are sound.
Eliminates toil through automation, reducing manual, repetitive operational work across the team.
Adds reliability guardrails to CI/CD pipelines (automated rollback, change-risk checks, progressive delivery) with the teams that own them.
Maintains production systems in accordance with applicable regulatory and compliance requirements, ensuring audit readiness and the ongoing effectiveness of access management, change management, logging, vulnerability remediation, patch management, and other operational controls.
Maintains documentation and evidence supporting business continuity and disaster recovery controls, including recovery testing results, remediation plans, and audit artifacts.
Partners with Security, Quality, and Compliance teams to maintain regulatory compliance, support internal and external audits, and drive timely resolution of audit findings and remediation activities.
Actively grows the capability of SRE and DevOps engineers, including consulting-partner engineers, through pairing, design and code review, and incident debriefs.
Creates conditions in which junior and contract engineers raise concerns and disagree openly, recognizing that silent agreement is a production risk.
Communicates in a documentation-first manner suited to a distributed, multi-time-zone team, so context is durable rather than held by one person.
Partners with software engineering, QA, and architecture to embed reliability and operability into the development lifecycle rather than only after incidents.
Informs capacity planning and scaling strategy with the Test team (who own load testing and modeling) and introduces proactive resilience testing such as game days once baseline support and observability are stable.
Partners with application, architecture, and business stakeholders to align business continuity and disaster recovery capabilities with application requirements and recovery objectives.
Supports cloud cost optimization: rightsizing, reserved capacity, and observability spend governance.
Ensures work complies with company policies, including Privacy/HIPAA and other regulatory, legal, and safety requirements. Other duties as assigned.