Sr. Site Reliability Engineer

O.C. Tanner•Salt Lake City, UT

1d•Hybrid

About The Position

O.C. Tanner is hiring a Senior Site Reliability Engineer to strengthen the reliability, scalability, and performance of its high-traffic, multi-tenant SaaS platform, including the e-commerce systems behind its recognition and awards programs. The role is part of a small SRE team supporting a large portfolio of applications. The team is moving toward an engineered approach based on SRE principles, including clear service ownership, SLOs, automation, and repeatable processes. The work begins with hands-on operational support and cleanup, with the goal of understanding recurring problems and engineering them out. The engineer will help define the team's future standards, tooling, and practices. This is a hybrid role based in Salt Lake City, Utah, with three days onsite and two days remote each week. The role participates in a regular on-call rotation, sharing follow-the-sun coverage with the team in India.

Requirements

At least 5 years of experience in Site Reliability Engineering, DevOps, or a related role.
Proficiency in at least one modern programming language, such as Python, Go, or Java.
Experience with Infrastructure-as-Code tools such as Terraform or CloudFormation.
Strong working knowledge of AWS and Kubernetes.
Hands-on observability experience with OpenTelemetry, Datadog, or similar platforms, including metrics, logging, tracing, and alerting strategies for distributed systems.
A track record of managing production incidents and reducing MTTR.

Nice To Haves

Experience with CI/CD pipelines and automation frameworks.
Experience with e-commerce or high-traffic, multi-tenant SaaS systems.

Responsibilities

Design, implement, and maintain monitoring, alerting, and tracing for cloud-native applications.
Define and track SLOs and error budgets, and improve uptime, latency, and overall system performance.
Manage observability tooling, currently including OpenTelemetry, Datadog, and Coralogix.
Work with engineering teams to define the metrics, logs, and traces needed for new and existing services.
Handle operational requests and support tickets from engineering teams, and automate recurring work.
Serve as incident commander for major incidents; lead root cause analysis and blameless postmortems, and drive action items to completion.
Improve incident detection, escalation, and resolution.
Manage and optimize Kubernetes clusters and AWS resources.
Automate infrastructure provisioning and scaling with Infrastructure-as-Code.
Partner with software engineering teams on reliability and monitoring requirements and help them adopt practices that improve resilience and performance.
Participate in a regular on-call rotation, respond to alerts, troubleshoot issues, and restore service during outages.
Reduce alert noise and improve the effectiveness of on-call over time.