Staff Software Engineer, Observability

Cerebras Systems • Sunnyvale, CA

6d

About The Position

Cerebras Systems is hiring a Staff Software Engineer to set technical direction for its observability platform. The role owns the platform’s long-term architecture and roadmap across metrics, logs, traces, and events, and drives adoption through RFCs, design reviews, and partnership with stakeholders.

The engineer will design telemetry pipelines for high-cardinality, high-frequency data generated by thousands of Wafer-Scale Engines, large Kubernetes clusters, and their storage and networking fabric. They will address ingestion and query performance challenges, partner across Engineering, SRE, Hardware, and ML Infrastructure to improve reliability and root-cause analysis, and bring hardware health signals into a unified observability layer.

The role also establishes OpenTelemetry instrumentation libraries and standards, promotes consistent observability practices across services, and mentors engineers while fostering operational excellence. Cerebras describes its work as building a breakthrough AI platform and supercomputer, with opportunities to publish and open-source AI research, alongside startup vitality and a non-corporate culture.

Requirements

• At least 8 years of software engineering experience.
• At least 4 years building or operating observability or monitoring platforms at significant scale, such as millions of active time series or petabytes of log data.
• Deep expertise in the open-source observability ecosystem, including experience operating, tuning, or extending systems such as Prometheus, Thanos, Cortex, Mimir, Elasticsearch, ClickHouse, or Loki, and understanding their internals, failure modes, and cost characteristics.
• Experience with OpenTelemetry instrumentation across a polyglot services environment.
• Strong Python experience; ability to write performant, concurrent systems code rather than only glue scripts.
• Strong distributed systems and Kubernetes expertise, including sharding, replication, consistency models, monitoring Kubernetes and its workloads, and data-pipeline reliability tradeoffs at scale.
• Experience with observability cost management and capacity planning at scale.
• A track record of setting technical direction and driving adoption across multiple teams.
• Ability to work effectively with ambiguity and translate business needs into technical strategy.

Nice To Haves

• Proficiency in Go.

Responsibilities

• Own the long-term architecture and roadmap for the observability platform across metrics, logs, traces, and events.
• Drive observability platform adoption across engineering through RFCs, design reviews, and stakeholder partnerships.
• Design telemetry pipelines capable of handling high-cardinality, high-frequency data from Wafer-Scale Engines, Kubernetes clusters, and associated storage and networking infrastructure.
• Solve large-scale telemetry ingestion and query performance problems.
• Partner with Engineering, SRE, Hardware, and ML Infrastructure teams to define SLOs, develop alerting strategies, and build tools that accelerate root-cause analysis and reduce MTTR.
• Make system health and failure diagnosis accessible to engineers across the organization.
• Work with engineering teams across the stack to surface critical hardware health signals in a unified observability layer alongside software metrics.
• Design and implement OpenTelemetry instrumentation libraries and standards, establishing conventions that scale across teams.
• Mentor senior engineers, influence engineering practices across team boundaries, and foster operational excellence.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed