Senior Software Engineer, Observability

Temporal Technologies

3d•$140,800 - $231,000

About The Position

Temporal Technologies is hiring a Senior Software Engineer on its Infrastructure Team, focused on internal and customer-facing observability. The role supports the productivity and reliability of Temporal’s developers and core platforms and works across the Temporal ecosystem to advance the organization’s mission. The engineer leads the full software development lifecycle, from gathering goals and requirements through design, implementation, deployment, operationalization, support, and maintenance. They build and operate distributed systems at scale, document design and operational knowledge, ensure suitable unit, integration, and performance testing, and take ownership of reliability and performance tradeoffs. The position also involves investigating issues, participating in the team’s on-call rotation, contributing to design reviews, and collaborating with stakeholders and users to develop effective features and tooling. The core observability technology stack includes ClickHouse, Prometheus, Grafana, Loki, Thanos, Kubernetes, and Temporal. The role calls for expertise in distributed systems, Kubernetes, query development, programming languages, cloud platforms, and infrastructure monitoring and instrumentation.

Requirements

Demonstrated ability to develop horizontally scalable, resilient, high-performance distributed systems in a production environment.
Experience designing, implementing, deploying, and supporting large-scale, geographically distributed observability systems, high-throughput data streaming or processing pipelines, or similar systems.
Expertise in one or more high-level programming languages.
Expert-level Kubernetes skills.
Expert-level query development skills.
Hands-on experience with one or more cloud providers.
Thorough understanding of computer architecture, operating systems, and networking.
Familiarity with best practices for monitoring, instrumenting, and configuring infrastructure.
User-first mindset and interest in empowering users through tooling.
Ability to work in a self-directed manner in a fast-paced environment.
Excellent collaboration and communication skills.

Nice To Haves

Proficiency in Go.
Experience with AWS or GCP.
Proficiency in SQL.

Responsibilities

Lead the end-to-end software development lifecycle, including requirements solicitation, design and review, implementation, deployment, operationalization, support, and maintenance.
Formulate feature designs, review them with stakeholders, incorporate feedback, and drive consensus.
Document design decisions and operational knowledge needed to deploy and manage developed software.
Provide appropriate unit, integration, performance, and production-readiness coverage for owned features.
Set a high bar for technical excellence and take ownership of developed software.
Design and build multi-component distributed systems that operate at scale.
Investigate issues methodically to identify root causes.
Evaluate performance and reliability implications of design options at scale and make related tradeoffs.
Participate in the team’s on-call rotation.
Develop expert knowledge of the architecture and services in the assigned domain and the Temporal ecosystem.
Investigate how Temporal’s software can best support the company’s mission.
Understand the needs of internal developers and external customers and apply that knowledge to product development and feature design.
Participate in design reviews and contribute to the design of other features.
Share design principles for building reliable systems at scale.