Principal Distributed Systems Engineer

Workday • Pleasanton, CA

2d • $187,100 - $334,300 • Hybrid

About The Position

The Principal Distributed Systems Engineer will own the technical vision and architecture for Workday’s data-product platform: a scalable lakehouse that helps teams build, govern, and operate data products and marts. This hands-on, high-autonomy role designs the platform end to end, spanning data modeling and dbt, orchestration, distributed query, table formats, catalogs, security, and infrastructure as code, while contributing and reviewing high-performance analytical SQL.

The engineer sets technical direction across multiple teams, mentors senior and staff engineers, and serves as the platform’s primary architect and escalation point. The platform stack includes dbt Core, Airflow, Trino, Apache Iceberg, Amazon S3 Tables, AWS Glue Catalog, access controls, and a visualization layer such as Apache Superset or Metabase. The role emphasizes scalable ingestion and interactive queries, reliability, governance, performance, cost efficiency, and developer experience.

The position is part of the Data Platform and Observability Engineering team. Workday’s flexible-work approach combines in-person and remote work, with at least half of each quarter spent in the office or in the field with customers, prospects, and partners, depending on the role. Remote home-office employees may also attend important in-person events.

Requirements

• 14 or more years of software development engineering experience.
• 6 or more years specifically designing, building, and operating complex distributed-system architectures, with successful deployment of highly available and fault-tolerant systems, such as systems with 99.9% uptime.
• 8 or more years of experience with at least two programming languages such as Java, Python, or Go, including production-level distributed-systems code.
• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Expert-level algorithmic thinking for efficient, scalable solutions to high-volume data and interactive query workloads.
• Deep expertise in API development, including advanced API protocols or architectural patterns for query engines, catalogs, and platform services.
• Deep understanding of distributed-systems principles, including partitioning, consensus or coordination, fault tolerance, and failure recovery.
• Proven ability to design and implement high-availability strategies for critical distributed data systems.
• Extensive experience with large-scale data processing and lakehouse platforms; examples include Kafka, Spark, Flink, Airflow, Trino, Apache Iceberg, Amazon S3 or S3 Tables, and AWS Glue Data Catalog.
• Production dbt Core experience at scale, including large model graphs, testing and data-quality frameworks, incremental models, and reusable project architecture.
• Strong production SQL and data-modeling experience, including query and physical-layout optimization.
• Experience using Terraform or an equivalent tool to design and operate production cloud infrastructure for a data platform.
• Experience designing multi-tenant access control, IAM, encryption, and governed access for analytical data.
• Experience with a visualization or semantic layer such as Apache Superset, Metabase, Looker, or comparable tools.
• Deep understanding of large-scale systems design, distributed data management, scalability, and cost-aware architecture.
• Strong understanding of security principles for complex distributed data environments.
• Proven ability to lead collaboration within and across software teams and drive architectural direction.
• Strong technical writing, documentation, and presentation skills.
• Still writes code and can start from a blank dbt project and a blank Terraform stack, explaining how Airflow, Trino, Iceberg, the catalog, object storage, and access control divide responsibilities.
• Treats the platform as a product and considers scalability, reliability, performance, security, operability, and developer experience.

Nice To Haves

• Master’s degree, such as an MS in Computer Science or Distributed Systems, is strongly preferred.
• Experience with visualization tools such as Apache Superset, Metabase, or equivalent; examples also include Looker.

Responsibilities

• Own the technical vision and architecture for Workday’s data-product platform and set technical direction across multiple teams.
• Architect and build a lakehouse platform using dbt Core, Airflow, Trino, Apache Iceberg, S3 Tables, and Glue Catalog for high-volume ingestion and interactive query performance.
• Own the platform path for shipping data products, including project structure, orchestration, CI/CD, environments, semantic and metrics definitions, and operational support.
• Optimize performance and scalability across data modeling and query workloads, including Iceberg layout and maintenance, partitioning, incremental processing, compression, and Trino tuning.
• Lead high-availability and disaster-recovery design for platform services, including multi-AZ resilience, failover, backup and restore, and appropriate recovery objectives.
• Design security and access-control architecture, including authentication and authorization for multi-tenant access across catalogs, compute, storage, and visualization.
• Use Terraform to define and evolve the platform, infrastructure, and environments, with attention to limiting blast radius.
• Own operational excellence through monitoring, alerting, capacity planning, safe reruns and backfills, and participation in the platform on-call rotation.
• Evaluate and introduce open-source and cloud-native technologies that materially improve scalability, cost efficiency, governance, or developer experience.
• Shape platform adoption through reusable dbt and modeling patterns, data-quality frameworks, and a visualization layer connected to governed catalogs and Trino.
• Publish best practices, mentor engineers across DPOE and partner teams, and act as an internal technical thought leader on modern lakehouse and data-product platforms.
• Operate autonomously in an ambiguous, fast-moving environment while aligning technical direction with broader platform strategy.
• Write and review production code and high-performance analytical SQL.

Benefits

• The role may be eligible for the Workday Bonus Plan or a role-specific commission or bonus.
• The role may be eligible for annual refresh stock grants.
• Workday describes a comprehensive benefits package but does not list specific benefits in this posting.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal