Senior Infrastructure Engineer, SRE

Rocket Money•San Francisco, WA

7d•Onsite

About The Position

We're looking to expand our Cloud Infrastructure team with a Senior Infrastructure Engineer, SRE to lead the reliability and operational evolution of our platform. We run hundreds of services in production, which enable us to process billions of transactions, consume multiple terabytes of data, and produce hundreds of millions of logs per day, and our reliability practice needs to evolve to match our growing scale. This includes: Building and improving the reliability and resiliency of our systems and services Establishing SLIs, SLOs, and error budgets for our most critical services and user journeys, and reviewing them regularly with the teams that own them Owning and evolving our disaster recovery strategy: recovery objectives, failover and restore paths, and regular exercises that prove they work Partnering with product engineering teams so they can own and operate their own services, with metrics that reflect real user experience Evolving our observability platform and standards across metrics, tracing, and logs: including instrumentation paved roads, alert quality, and observability cost Strengthening our incident practice: tuning paging thresholds, keeping runbooks current, and following through on postmortem action items Contributing to day-to-day Cloud Infrastructure work alongside your reliability specialty — infrastructure build-outs, platform backlog, and a shared on-call rotation (1 week out of every 6 weeks). You'll join the Cloud Infrastructure team and partner with engineering and internal support teams to drive this work. We support millions of people to improve their financial lives, and this role ensures we can continue to do so reliably and at scale.

Requirements

5+ years of hands-on cloud or infrastructure engineering experience, with substantial time spent on reliability and production operations at scale
Defined SLIs and SLOs for real production services, and can talk about what changed as a result. What got fixed, what got deprioritized, and what you got wrong the first time
Hands-on experience with an observability platform in production; Datadog strongly preferred
Comfortable writing code (Python, Go, TypeScript, or similar) for internal tooling, production debugging, and automation
Write production Terraform and are comfortable in AWS, and when production breaks you can find the problem and fix it
Built or operated a disaster recovery plan: you set the recovery goals, wrote the failover and restore steps, and ran the drills that proved it works
Been on-call for services you helped build, and you have opinions about what makes an alert worth waking someone for
Prefer giving teams paved roads and good defaults over mandates, so they can own their own instrumentation

Nice To Haves

Led a reliability or observability modernization project where you defined the vision, approach, and delivered the implementation
Built internal tooling, libraries, or instrumentation standards that made it easier for other teams to operate their services well
Run game days, chaos experiments, or DR exercises, and fixed the problems they uncovered
Cut observability spend while keeping the coverage you needed

Responsibilities

Building and improving the reliability and resiliency of our systems and services
Establishing SLIs, SLOs, and error budgets for our most critical services and user journeys, and reviewing them regularly with the teams that own them
Owning and evolving our disaster recovery strategy: recovery objectives, failover and restore paths, and regular exercises that prove they work
Partnering with product engineering teams so they can own and operate their own services, with metrics that reflect real user experience
Evolving our observability platform and standards across metrics, tracing, and logs: including instrumentation paved roads, alert quality, and observability cost
Strengthening our incident practice: tuning paging thresholds, keeping runbooks current, and following through on postmortem action items
Contributing to day-to-day Cloud Infrastructure work alongside your reliability specialty — infrastructure build-outs, platform backlog, and a shared on-call rotation (1 week out of every 6 weeks)