Principal Software Engineer, Orchestration

Roblox•San Mateo, CA

3d•$295,250 - $345,040•Onsite

About The Position

Roblox’s data platform powers analytics, machine learning, and product decisions across a platform serving more than 200 million daily active users. The Principal Software Engineer will serve as the technical leader for the Data Platform Orchestration team, which owns Apache Airflow and the Kubernetes (EKS) infrastructure that runs more than 8,000 DAGs daily, most of them Spark jobs. The role sets the long-term architecture for data orchestration at Roblox and improves its speed, reliability, and usability for hundreds of engineers. The position defines the multi-year architecture for Airflow and its surrounding platform, including scheduler and executor design, multi-tenancy, upgrades, migrations, and future platform direction. It also leads the design and operation of EKS clusters for orchestration and Spark workloads, and builds APIs, operators, providers, libraries, and self-service tools for internal data teams. The Principal Engineer establishes reliability practices such as SLOs, alerting, diagnostics, and incident reviews, addressing root causes in code. The role works across data platform, governance, data science, and product engineering teams to align roadmaps and technical decisions, while setting engineering standards and mentoring engineers. Office-based roles are onsite Tuesday through Thursday, with optional presence Monday and Friday. The listed base salary range applies to roles based at Roblox headquarters in San Mateo, California.

Requirements

B.S. equivalent in computer science or sufficient experience.
At least 10 years of experience designing, building, and operating production-grade, large-scale distributed systems.
Deep hands-on experience designing, operating, and scaling a production data workflow orchestrator or job scheduler handling thousands of workflows or DAGs per day; Apache Airflow is ideal, and equivalent systems are also accepted.
Understanding of scheduler and executor design, metadata store performance, task isolation, retries and idempotency, and safe upgrades and migrations.
Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling with Karpenter or Cluster Autoscaler, and GitOps-style deployment.
Strong proficiency in Python and proficiency in Go or Java, with a record of writing clean, well-tested platform and tooling code.
Working knowledge of how Spark jobs run on Kubernetes, such as Spark on K8s, spark-operator, or EMR on EKS, and how orchestration interacts with them.
Experience building platforms used by other engineers, with strong instincts for APIs, observability, reliability, and developer experience.
Track record of influencing technical direction across a large engineering organization and driving multi-quarter projects from concept to production.
Ability to define problems when requirements are unclear, unblock dependencies, and carry complex work through to stable production.

Nice To Haves

Contributions to Apache Airflow or related open-source projects, such as Airflow providers, Helm chart, spark-operator, Karpenter, or Argo CD.
Experience with other orchestration systems such as Dagster, Temporal, Argo Workflows, or Flyte, and judgment about when each is appropriate.
Experience with Terraform or other infrastructure-as-code, Prometheus or OpenTelemetry, and cost optimization or chargeback for shared platforms.
Experience operating infrastructure at consumer-internet scale.

Responsibilities

Own the multi-year architecture and strategy for Airflow and the surrounding orchestration platform, including scheduler and executor design, multi-tenancy, upgrades, migrations, and future platform direction.
Identify and remove scaling limits across the scheduler, metadata database, executors, and DAG parsing to ensure predictable latency and strong SLAs for thousands of daily DAGs.
Lead the design and operation of EKS clusters for orchestration and Spark workloads, including autoscaling, resource isolation, cost efficiency, and deployment automation.
Design APIs, custom operators and providers, libraries, and self-service tooling that enable data teams to build, deploy, and monitor pipelines safely.
Establish SLOs, alerting, and diagnostics for pipelines and platform components; lead incident reviews and address root causes in code.
Collaborate with data platform, governance, data science, and product engineering teams to align roadmaps, guide technical decisions, and build consensus on complex initiatives.
Set standards for code quality, design, and operational readiness.
Mentor staff, senior, and mid-level engineers.