Principal Distributed Systems Engineer

Workday • Pleasanton, CA

1d • Hybrid

About The Position

Workday’s Data Platform and Observability Engineering (DPOE) team is building a next-generation data-product platform: the compute, storage, catalog, orchestration, governance, and developer workflows engineering and analytics teams use to create and operate production data products and marts. The platform is based on a modern lakehouse stack that includes dbt Core, Airflow, Trino, Apache Iceberg, Amazon S3 Tables, AWS Glue Catalog, access control, and a visualization layer such as Apache Superset or Metabase.

The Principal Distributed Systems Engineer owns the technical vision and architecture for this highly scalable platform. This is a hands-on, high-autonomy role that designs the platform end to end, writes and reviews performant analytical SQL, sets technical direction across teams, mentors senior and staff engineers, and serves as the primary architect and escalation point. The role covers project and pipeline design, query performance, governance, reliability, and operational support.

The engineer will guide how teams build, govern, and operate data products and marts, while improving scalability, reliability, performance, security, operability, cost efficiency, and developer experience. They will work with minimal oversight while aligning technical decisions with the broader platform strategy.

Requirements

• 14 or more years of experience in software development engineering.
• 6 or more years specifically designing, building, and operating complex distributed-system architectures, with successful deployment of highly available systems (for example, 99.9% uptime) and fault tolerance.
• 8 or more years of experience with at least two programming languages such as Java, Python, or Go, including production-level distributed-systems code.
• Bachelor’s degree in a relevant field such as Computer Science, Engineering, or a related discipline, or equivalent practical experience.
• Expert-level algorithmic thinking for efficient, scalable solutions to high-volume data and interactive-query workloads.
• Deep expertise in API development, including advanced API protocols or architectural patterns around query engines, catalogs, and platform services.
• Deep understanding of distributed-systems principles, including partitioning, consensus or coordination, fault tolerance, and failure recovery.
• Proven ability to design and implement high-availability strategies for critical distributed data systems.
• Extensive experience with large-scale data processing and lakehouse platforms, such as Kafka, Spark, Flink, Airflow, Trino, Apache Iceberg, Amazon S3 or S3 Tables, and AWS Glue Data Catalog.
• Production dbt Core experience at scale, including large model graphs, testing and data-quality frameworks, incremental models, and reusable project architecture.
• Strong production SQL and data-modeling experience, including query and physical-layout optimization.
• Experience using Terraform or an equivalent to design and operate production cloud infrastructure for a data platform.
• Experience designing multi-tenant access control, IAM, encryption, and governed access for analytical data.
• Experience with a visualization or semantic layer such as Apache Superset, Metabase, Looker, or comparable tools.
• Deep understanding of large-scale systems design, including distributed data management, scalability, and cost-aware architecture.
• Strong understanding of system security principles for complex distributed data environments.
• Proven ability to lead collaboration within and across software teams and drive architectural direction.
• Strong technical writing, documentation, and presentation skills.

Nice To Haves

• Master’s degree, such as an MS in Computer Science, Distributed Systems, or a related field, is strongly preferred.

Responsibilities

• Architect and build the data-product platform using dbt Core, Airflow, Trino, Apache Iceberg, S3 Tables, and Glue Catalog for high-volume ingestion and interactive query performance.
• Own the platform workflows teams use to ship products, including project structure, orchestration, CI/CD, environments, semantic and metrics definitions, and operational support.
• Improve performance and scale across modeling and query paths through Iceberg layout and maintenance, partitioning, incremental processing, compression, and Trino tuning under production load.
• Lead high-availability and disaster-recovery design for platform services, including multi-AZ resilience, failover, backup and restore, and appropriate recovery objectives.
• Design security and access-control architecture, including authentication and authorization for multi-tenant data access across catalog, compute, storage, and visualization.
• Use Terraform to define and evolve the platform, setting standards for infrastructure, environments, and blast radius.
• Own operational excellence, including monitoring, alerting, capacity planning, safe reruns and backfills, and participation in the platform on-call rotation.
• Evaluate and introduce open-source and cloud-native technologies that materially improve scalability, cost efficiency, governance, or developer experience.
• Shape platform usage through reusable dbt and modeling patterns, data-quality frameworks, and a visualization layer connected to governed catalogs and Trino.
• Publish best practices, mentor engineers across DPOE and partner teams, and act as a technical thought leader for the modern lakehouse and data-product stack.
• Set technical direction with minimal oversight in a fast-moving, ambiguous environment while aligning with broader platform strategy.
• Write and review highly performant analytical SQL and production code.

Benefits

• May be eligible for the Workday Bonus Plan or a role-specific commission/bonus.
• May be eligible for annual refresh stock grants.
• Workday describes its comprehensive benefits but does not enumerate them in the posting.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal