Staff Software Engineer

Temporal Technologies

7h•$185,000 - $270,000

About The Position

Temporal is an open source programming model that can simplify code, make applications more reliable, and help developers focus on the important things like delivering features faster. We are on a mission to be the reliable foundation of every developer’s toolbox, and are building the team that will make that happen. Our values guide us —they are present in how we show up, make decisions, and work together to make an impact. We’re curious, driven, collaborative, genuine and humble. Temporal is growing and we are looking for those who share our values, challenge 'standard' thinking, and want to influence our future. If you have a passion for improving the developer experience, building world-class open-source software and communities, and want to be a part of our amazing team, we'd love to hear from you! Staff Engineer – Cloud Data Store (Visibility) Temporal is an open-source programming model that can simplify code, make applications more reliable, and help developers focus on the important things like delivering features faster. We are on a mission to be the reliable foundation of every developer's toolbox, and are building the team that will make that happen. Our values guide us — they are present in how we show up, make decisions, and work together to make an impact. We're curious, driven, collaborative, genuine, and humble. Temporal is growing and we are looking for those who share our values, challenge "standard" thinking, and want to influence our future. If you have a passion for improving the developer experience, building world-class open-source software and communities, and want to be part of our amazing team, we'd love to hear from you! ⸻ Summary Cloud Data Store (CDS) owns the storage, retrieval, and lifecycle of all workflow data at planet scale. We design the persistence APIs, build storage abstractions that run across cloud vendors, and deliver the observability that lets customers trust their state machines for years. [To see more detail re: the Temporal CDS Eng team, click here [new window] A core part of CDS is Temporal Visibility: the system that powers workflow listing, filtering, search, and observability across massive, long-lived state machines. Visibility has demanding characteristics — high write throughput, complex secondary indexing, low-latency queries, strict correctness guarantees, and the need to operate continuously while schemas and backends evolve. As a Staff Engineer on CDS, you will take a leading role in a ground-up rewrite of the Visibility persistence layer, with the goal of making it dramatically more performant, scalable, and operable. This includes evaluating and driving deep integrations with modern analytical and search data stores such as ClickHouse and Elasticsearch, with a strong preference for self-hosted operation (while supporting managed offerings where appropriate). You will design, build, and operate significant portions of our backend for highly scalable, multi-tenant services, and own large technical initiatives end-to-end — from initial design through live data migration, rollout, and long-term operational stewardship. ⸻

Requirements

5 or more years of experience as an "Arranger" and/or "Builder/Enhancer" of highly scalable distributed systems. see HERE for more info re: "Arranger" and/or "Builder/Enhancer".
Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes
Significant experience writing and operating concurrent production systems in Go, Java, or similar languages, at a high-end intermediate to expert level
Experience writing concurrent code in production with languages like Go or Java or other applicable languages with skill level as "high end of Intermediate" and/or "Advanced" or "Expert" levels. see HERE for more info re: "high end of Intermediate" and/or "Advanced" or "Expert levels"
Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments (managed services are a strong plus).
Experience building and running services on AWS. Bonus: Azure and/or GCP experience.
Demonstrated ability to lead large, multi-quarter technical initiatives, especially those involving core data infrastructure and live data migrations.

Nice To Haves

Prior contributions to Temporal, Cadence, or other workflow engines.
Deep expertise in storage internals (e.g., columnar stores, LSM trees, inverted indexes, transactional logs).
Experience operating multi-region services with ≥99.99% uptime.
Strong background in operating and evolving Open Source systems.
Experience building Kubernetes controllers and/or CRDs.

Responsibilities

Re-architect Temporal Visibility at scale
Lead the design and implementation of a new persistence layer for Temporal Visibility, informed by its real-world access patterns (high-volume writes, time-based queries, filtering, sorting, and pagination across long-running workflows).
Evaluate and select the most appropriate storage technologies (e.g., ClickHouse, Elasticsearch, or complementary systems), clearly articulating tradeoffs around indexing models, consistency, cost, latency, and operational complexity.
Design schemas, APIs, and query models that make Visibility both powerful and intuitive for customers.
Deliver safe, large-scale data migrations
Plan and execute online migrations of live Visibility data from existing persistence stores to new backends, at scale and without customer downtime.
Design dual-write, backfill, validation, and cutover strategies that prioritize correctness, observability, and rollback safety.
Build tooling and automation to validate data integrity and performance throughout the migration lifecycle.
Own performance, reliability, and operability
Define and own SLOs for Visibility storage and query paths.
Profile hot paths, design benchmarks, and lead systematic performance tuning efforts.
Build operational playbooks, dashboards, and alerting that make the system understandable and debuggable for on-call engineers.
Lead incident reviews and reliability improvements related to persistence and indexing systems.
Technical leadership and collaboration
Break down large, ambiguous roadmap initiatives into concrete, executable phases.
Author and steward design docs and RFCs through review with peers and stakeholders.
Mentor and unblock other engineers working in the persistence and storage domain.
Partner closely with Server, Cloud, and Developer Experience teams to land features end-to-end.

Benefits

Unlimited PTO, 12 Holidays + 2 Floating Holidays
100% Premiums Coverage for Medical, Dental, and Vision
AD&D, LT & ST Disability, and Life Insurance (Standard & Supplemental Available)
Empower 401K Plan
Additional Perks for Learning & Development, Lifestyle Spending, In-Home Office Setup, Professional Memberships, WFH Meals, Internet Stipend and more!
Paid Time Off (PTO) and Benefits outside the United States vary by country, and are issued in partnership with Remote.com. Additionally, Temporal offers perks to all international employees for learning & career development, a lifestyle spending account, in-home office setup (in addition to company-issued hardware), professional memberships, work-from-home meals, and access to the Calm app for mental wellness.
$3,600 / Year Work from Home Meals
$1,800 / Year Professional Enrichment (Career Development & Professional Memberships)
$1,200 / Year Lifestyle Spending Account
$1,000 / Year In-Home Office Setup (In addition to Temporal issued equipment - laptop, monitor, keyboard, mouse, trackpad, and extension power cable at no cost to you)
$74 / Month Reimbursement for Internet
Calm App Subscription for Mental Health & Wellness