Principal SRE

Backblaze External Website

8h•$215,000 - $250,000

About The Position

Backblaze is seeking a Principal Site Reliability Engineer to help ensure the stability, scalability, and reliability of production systems. The role involves collaborating across functions to design, implement, and manage core infrastructure, with a long-term strategic focus on platform stability. As a technical leader, the Principal SRE will work across the organization, including with executives and customers, and use observability, process improvements, AI, and automation to mitigate issues faster and meet customer demand. The Principal SRE owns reliability outcomes for critical production services, contributes to service health metrics and reliability policies, participates in leadership on-call and incident processes, and drives improvements from discovery through implementation. The role also designs operational automation and monitoring frameworks, collaborates on storage design and performance expectations, and supports resilient system design with engineering, product, customer success, operations, vendors, and customers. Backblaze is an open-cloud storage company founded in 2007 and publicly traded on Nasdaq. It reports more than $136 million in ARR, manages over three billion gigabytes of data, and serves more than 500,000 customers in over 175 countries.

Requirements

Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience.
10+ years of experience in site reliability, systems engineering, or operations.
Experience with large-scale, production-grade systems, including establishing operational procedures or runbooks and bringing others up to speed on them.
Solid Linux systems administration and troubleshooting skills.
Proven ability to leverage AI tools to improve platform stability and scalability.
Knowledge of storage systems, performance, sharding, and storage algorithms.
Experience with load-balancing services, including IPVS.
Security-minded operations experience, including patching, hardening, and identifying vulnerabilities.
Familiarity with reliability concepts such as monitoring, alerting, incident response, and root cause analysis.
Proficiency in at least one scripting language, such as Python, Bash, or Go.
Experience with containers (Kubernetes, Docker) and microservices concepts.
Comfort operating in a Kubernetes/Vitess-on-K8s environment, including kubectl, mysqlsh, and Vitess keyspaces.
Experience with HashiCorp products such as Nomad, Vault, or Terraform.

Nice To Haves

Experience with hyperscalers.
Experience in a SaaS, service provider, or distributed systems environment.
Familiarity with ITIL/OSS practices and SLOs/SLAs.
Strong problem-solving skills and willingness to learn new technologies.
Experience with cloud platforms such as AWS, GCP, or Azure.

Responsibilities

Own the availability and durability of critical services across production environments and be accountable for outcomes.
Work independently, take ownership, and drive projects from problem discovery through resolution.
Assist and advise on service health metrics using SLIs, SLOs, and error budgets; partner with service owners to define and iterate on these measures and establish error-budget policies and alerting.
Escalate issues when service health thresholds are at risk.
Participate in leadership on-call rotations, incident response, and post-incident reviews to drive service improvements.
Follow established ITIL/OSS processes for incident, change, problem, and capacity management.
Design automation for common operational tasks to reduce manual intervention and toil.
Collaborate with engineering on storage design and performance expectations against product SLAs.
Define monitoring, logging, and alerting frameworks and help integrate runbooks with FireHydrant.
Work with CI/CD pipelines, configuration management, and infrastructure-as-code tools.
Write scripts to improve system reliability and efficiency.
Partner with engineering, customer success, product, and operations teams to support resilient system design and operations.
Work with customers to prioritize stability based on their needs and product experiences.
Work with vendors and service providers to troubleshoot service issues and track SLA performance.
Document systems, share learnings, and help grow a reliability-minded engineering culture.
Lead cross-functional teams to identify gaps in observability and operational processes and define prioritization strategies.
Identify recurring issues, propose long-term improvements, and design implementation action plans.
Promote reliability-focused practices within development and operations teams.