Sr. Observability Systems Engineer

Backblaze External Website

1d•$178,000 - $200,000

About The Position

This is a senior individual contributor role with significant architectural ownership. The engineer will set technical direction for how Backblaze detects, correlates, and responds to infrastructure incidents across its data centers, and will mentor engineers as the observability platform matures. The role focuses on making complex distributed systems visible and applying AI and agentic automation to infrastructure operations. The position owns and evolves metrics, logging, and distributed tracing platforms; establishes dashboard and instrumentation standards; and develops telemetry capacity and retention strategies. It also owns infrastructure-as-code for observability deployments across Kubernetes clusters, automates configuration across Debian-based bare-metal servers, and partners with SRE, Storage Engineering, Network Engineering, and application teams. The work supports infrastructure spanning multiple data centers, including Kubernetes, Ceph, Vitess, KVM, and a custom storage platform serving hundreds of thousands of customers. The expected US base salary range is $178,000–$200,000, plus a 10% target annual bonus and RSUs. Final offers depend on factors including candidate location, skills, work experience, and relevant credentials. Backblaze describes its workplace as collaborative and values engineering excellence, transparency, operational rigor, diversity, equity, and inclusion.

Requirements

At least 8 years in systems engineering, SRE, DevOps, or an observability-focused role, including demonstrated technical leadership on a platform or infrastructure team.
Deep hands-on experience with Prometheus, VictoriaMetrics, or equivalent metrics platforms, including PromQL, Alertmanager, and operating these systems at scale.
Experience designing and operating log aggregation pipelines such as Elasticsearch/OpenSearch, Logstash, Filebeat, or similar.
Working knowledge of distributed tracing, including instrumentation strategy, with Jaeger, OpenTelemetry, or equivalent.
Proficiency in Python and/or Go for automation and tooling.
Experience with Kubernetes, including deploying workloads and understanding pod networking and storage.
Experience with infrastructure-as-code tooling and configuration management.
Strong understanding of Linux internals; Debian/Ubuntu is preferred.
Demonstrated experience mentoring engineers and raising a team's technical bar.
Excellent written and verbal communication skills, including writing design documents that drive decisions and working across engineering, SRE, security, and leadership.

Nice To Haves

Experience leading evaluation or deployment of AIOps tooling, including alert correlation, deduplication, or automated root cause analysis.
Experience building, deploying, or operating AI agents or agentic AI systems.
Familiarity with LLM APIs such as Anthropic or OpenAI, or self-hosted LLM inference in an infrastructure context.
Familiarity with Ceph storage, including OSD management, pool configuration, and CRUSH maps.
Experience with incident management platforms.
Familiarity with BGP, network observability, or multi-data-center environments.
Experience with secrets management in production environments.
Experience operating in a regulated or compliance-driven environment.
Prior experience at a storage-focused company or in a hyperscale infrastructure environment.

Responsibilities

Set organization-wide technical direction and standards for observability, including instrumentation conventions, alerting philosophy, and dashboard design patterns.
Mentor Observability team engineers and serve as a technical resource for SRE, Storage Engineering, and Network Engineering.
Lead observability design reviews and represent the team in cross-functional architecture discussions.
Evaluate build-versus-buy decisions for observability and AIOps tooling, including data sovereignty and compliance considerations.
Produce and maintain design documentation, runbooks, and architectural decision records.
Own and evolve the metrics pipeline, including VictoriaMetrics, Prometheus, and Consul-based service discovery for scrape targets.
Own and expand the OpenSearch, Logstash, and Filebeat logging stack across Backblaze data centers.
Architect and expand distributed tracing with Jaeger and OpenTelemetry, extending instrumentation to additional services.
Define and maintain Grafana dashboard standards for infrastructure, storage, and application performance visibility.
Expand internal application telemetry in partnership with application teams.
Define telemetry-stack capacity and retention strategies as the fleet grows.
Own infrastructure-as-code for observability deployments across multiple Kubernetes clusters.
Write and review automated playbooks for configuration management and automation across Debian-based bare-metal servers.
Partner with SRE on Consul-based service discovery for VictoriaMetrics scrape-target automation.
Lead Ceph storage observability, including OSD health, performance metrics, and alerting integration.
Collaborate with Network Engineering on BGP, routing, and network-level observability.