Principal Engineer, Data Platform

Traversal•New York, NY
•$275,000 - $350,000•Onsite

About The Position

Traversal is hiring a Principal Engineer, Data Platform, to own the technical direction of its petabyte-scale data platform and Production World Model. The platform supports the Traversal AI SRE agent, processing very large volumes of production data so the agent can investigate incidents using timely, relevant information. The role defines how data is ingested, stored, queried, and transformed across SaaS and BYOC deployments in demanding enterprise environments. This is a hands-on technical leadership position combining architecture, roadmap development, design documentation, and production coding. The Principal Engineer will work closely with company leadership, engineering, Product, Forward Deployed Engineering, Go-to-Market, and AI Research, and will participate in architecture reviews and deployments with large enterprise customers. The role also involves mentoring engineers and establishing standards used across teams. The position is full-time and in-person, based in New York near Madison Square Park. The stated U.S. base salary range is $275,000 to $350,000, plus equity and benefits.

Requirements

  • Typically 10 or more years of senior-level experience building and operating production data infrastructure, including setting architecture for a data platform used by multiple teams.
  • Deep distributed-systems expertise in streaming data, analytical storage, and query engines, including Kafka or equivalent; ClickHouse, Doris, Druid, or Pinot; data lakehouse technologies; and scalable, reliable multi-tenant systems.
  • Strong systems-programming skills in Rust or comparable experience in Go, plus comfort with Python.
  • Experience shipping on Kubernetes, Helm, Terraform, and AWS, ideally in environments not fully controlled by the company, such as BYOC, on-premises, or regulated customer clouds.
  • A record of leading multi-quarter migrations and platform transformations through adoption and deprecation.
  • Regularly write production code and be able to discuss recent work personally shipped.
  • Sound cost judgment and the ability to evaluate performance, cost, durability, and simplicity tradeoffs using real numbers.
  • Clear technical writing and judgment, including the ability to make crisp tradeoff decisions, build alignment without authority, and contribute effectively in architecture reviews with enterprise customers.

Nice To Haves

  • Experience building the backend of an observability, logging, APM, or SIEM product.
  • Fluency with OpenTelemetry/OTLP and the data models and APIs of major observability vendors such as Datadog, Splunk, Elastic, Prometheus, and Dynatrace.
  • Experience with workflow orchestration tools such as Airflow, Dagster, or Temporal, and data pipeline frameworks.
  • Experience building LLM-powered data pipelines for enrichment, summarization, or embeddings, including cost control.
  • Background in data governance, including PII redaction, data residency, or regulated industries such as financial services.
  • Contributions to open-source data infrastructure.

Responsibilities

  • Own the data platform architecture across direct-push ingestion, scheduled pulls from third-party observability platforms, non-telemetry data, and transformation pipelines.
  • Set the storage and query strategy, including tradeoffs among time-series, columnar/OLAP, and lakehouse technologies, retention, cost, and agent-query performance.
  • Design query interfaces, performance and freshness guarantees, and safe sandboxing for the AI agent's ad-hoc queries.
  • Partner with AI Research to provide investigation history, evaluation datasets, and derived signals that improve the agent and its research loops.
  • Work with engineering and company leadership on the technical roadmap, and bring customer and cross-functional input into platform decisions.
  • Lead the orchestration and transformation framework for scheduled, idempotent, observable jobs that ingest non-telemetry context and produce derived data such as materialized views, rollups, entity graphs, and cached LLM-based enrichment.
  • Own reliability, performance, and cost end to end; define and meet SLOs for availability, data freshness, and query latency; provide monitoring, runbooks, and safe rollouts; and lead technical incident response.
  • Design customer-data controls for redaction, tenant isolation, and data residency.
  • Mentor senior engineers and establish standards and reference implementations adopted by teams beyond Data Platform.
  • Write production code and lead multi-quarter migrations and platform transformations through adoption and deprecation.

Benefits

  • Competitive compensation
  • Startup equity
  • Health insurance
  • Fertility benefits
  • Technology setup stipend
  • Flexible time off
  • In-office snacks
  • Team happy hours and outings
  • Annual company offsite
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service