Sr. Software Engineer, Observability and Telemetry

Tenstorrent•Santa Clara, CA

2d•$100,000 - $500,000•Hybrid

About The Position

Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance expectations, ease of use, and cost efficiency. With AI redefining the computing paradigm, solutions must evolve to unify innovations in software models, compilers, platforms, networking, and semiconductors. Our diverse team of technologists have developed a high performance RISC-V CPU from scratch, and share a passion for AI and a deep desire to build the best AI platform possible. We value collaboration, curiosity, and a commitment to solving hard problems. We are growing our team and looking for contributors of all seniorities. Tenstorrent is building the world’s fastest, most efficient AI compute clusters. Our modular RISC-V and AI processors can snap together into a single, massively parallel distributed supercomputer consisting of thousands of compute nodes. As we scale, the volume and complexity of operational data grows by orders of magnitude. Observability and telemetry are key to ensuring our customers can resolve problems in minutes rather than hours. The telemetry team owns our proprietary telemetry infrastructure, spanning from the device level to the infrastructure needed to drive dashboards, monitoring systems, and orchestration. This role is hybrid, based out of Santa Clara, CA; Austin, TX; or Toronto, ON. We welcome candidates at various experience levels for this role. During the interview process, candidates will be assessed for the appropriate level, and offers will align with that level, which may differ from the one in this posting.

Requirements

Strong C++ engineer and comfortable working in both low-level environments and distributed systems design.
Experience building atop observability platforms such as Prometheus, OpenTelemetry, Grafana, ClickHouse, or similar technologies.
Solid understanding of data structures for manipulating large volumes of data.
Familiarity with SQL databases, with time-series databases a plus.
Curious about networking and communication across large clusters and comfortable reasoning from first principles while challenging industry conventions.

Responsibilities

Architect, implement, and maintain TT-Telemetry, our C++-based service for collecting and exporting device-level metrics.
Interface with internal engineering teams to build a deep understanding of Tenstorrent’s architecture and identify and surface useful metrics.
Design efficient built-in web GUIs for observing device- and cluster-level state, diagnosing problems, and monitoring utilization.
Design ingestion pipelines for industry standard telemetry systems (e.g., Prometheus).
Help define the long-term architecture of Tenstorrent’s distributed telemetry stack.