Staff Software Engineer, Observability

Cerebras Systems • Sunnyvale, CA

1d

About The Position

Cerebras Systems is hiring a Staff Software Engineer to set the technical direction for its observability platform. The role owns the platform’s long-term architecture and roadmap across metrics, logs, traces, and events, and drives adoption through RFCs, design reviews, and collaboration with stakeholders.

The engineer will design telemetry pipelines for high-cardinality, high-frequency data from Wafer-Scale Engines, Kubernetes clusters, and the connected storage and networking fabric. The role also advances reliability across the organization by partnering with Engineering, SRE, Hardware, and ML Infrastructure teams on SLOs, alerting, and root-cause analysis tools. It includes connecting hardware health signals with software metrics, establishing OpenTelemetry instrumentation libraries and standards, and mentoring engineers while fostering operational excellence.

Requirements

• At least 8 years of software engineering experience.
• At least 4 years building or operating observability or monitoring platforms at significant scale, including millions of active time series and petabytes of log data.
• Deep expertise in the open-source observability ecosystem.
• Experience operating, tuning, and extending systems such as Prometheus, Thanos, Cortex, Mimir, Elasticsearch, ClickHouse, or Loki, including understanding their internals, failure modes, and cost characteristics.
• Experience with OpenTelemetry instrumentation in a polyglot services environment.
• Strong Python experience; ability to write performant, concurrent systems code.
• Strong distributed systems and Kubernetes expertise, including sharding, replication, consistency models, monitoring Kubernetes and its workloads, and the tradeoffs involved in reliable data pipelines at scale.
• Experience with observability cost management and capacity planning at scale.
• Track record of setting technical direction and driving adoption across multiple teams.
• Ability to work effectively amid ambiguity and translate business needs into technical strategy.

Nice To Haves

• Proficiency in Go.

Responsibilities

• Own the long-term architecture and roadmap for the observability platform across metrics, logs, traces, and events.
• Drive observability platform adoption through RFCs, design reviews, and partnership with stakeholders.
• Design telemetry pipelines for high-cardinality, high-frequency data from Wafer-Scale Engines, Kubernetes clusters, and their storage and networking fabric.
• Solve ingestion and query performance problems that challenge conventional approaches.
• Partner with Engineering, SRE, Hardware, and ML Infrastructure teams to define SLOs, build alerting strategies, and create tools that accelerate root-cause analysis and reduce MTTR.
• Make it easier for engineers to understand system health and diagnose failures.
• Work with engineering teams across the stack to surface hardware health signals in a unified observability layer alongside software metrics.
• Design and implement OpenTelemetry instrumentation libraries and standards, and establish conventions that scale across teams.
• Mentor senior engineers, influence engineering practices across teams, and foster operational excellence.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed