Senior Software Engineer, Spark Platform

DoorDash•San Francisco, NY

17h•Hybrid

About The Position

The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem, including the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers data, analytics, and machine-learning workloads. The team operates Spark at significant scale and addresses distributed-systems challenges across thousands of cluster deployments through runtime optimization, systems architecture, multi-tenant scheduling, and end-user tooling. As a Senior Software Engineer, you will set the technical direction and shape the architecture of DoorDash's in-house Spark deployment. You will own cross-cutting challenges spanning runtime, shuffle, scheduling, and service reliability, making architectural decisions for the platform's long-term evolution. You will partner with the Engineering Manager on the technical roadmap, hiring, interview design, and team shape; mentor engineers; and represent Spark Platform in partnerships and architecture forums with Data Engineering, ML Platform, and product engineering teams. The role is hybrid and requires the employee to be located in San Francisco, Sunnyvale, Seattle, or New York City. It reports to the Engineering Manager on the Spark Platform team.

Requirements

B.S., M.S., Ph.D. in Computer Science, or equivalent.
At least 6 years of industry experience designing and operating distributed systems at scale.
Deep, hands-on experience with Apache Spark internals, query execution, shuffle, and the executor/driver model at platform scale, on Amazon EMR, Databricks, or an in-house deployment. Experience should focus on platform operations rather than authoring individual Spark jobs.
Production experience with one or more of remote or external shuffle systems, batch or big-data schedulers, or observability and SRE practices for operating distributed compute platforms.
Strong experience operating production Kubernetes workloads, including operator patterns, executor pod lifecycle, network topology, and multi-tenant failure modes.
Professional experience with Scala, Java, Python, or Go, and strong SQL skills.
Located or willing to relocate to the Bay Area, Seattle, or New York City.

Nice To Haves

Familiarity with data lake table formats such as Apache Iceberg or Delta Lake, and with the query and SQL engines that read them.
Track record of technical leadership on a platform team, including setting direction, mentoring, and partnering with management on roadmap and hiring.
Experience with remote or external shuffle systems such as Celeborn, Magnet, or Cosco.
Experience with batch or big-data schedulers such as YuniKorn, Volcano, Kueue, or the Spark-on-Kubernetes operator.

Responsibilities

Set the multi-year technical direction for the in-house Spark-on-Kubernetes platform, including its runtime, shuffle, scheduler, and reliability.
Own deep distributed-systems challenges involving shuffle architecture, multi-tenant scheduling, runtime performance, and large-scale failure modes.
Partner with the Engineering Manager on the technical roadmap, hiring, interview design, and team shape.
Uplevel the team through design reviews and mentorship, and raise the quality bar for shipped work.
Represent Spark Platform in cross-team architecture forums and shape how data, analytics, and ML workloads use the platform.

Benefits

401(k) plan with employer matching.
16 weeks of paid parental leave.
Wellness benefits.
Commuter benefits match.
Paid time off and paid sick leave in compliance with applicable laws.
Equity grant opportunities.
Medical, dental, and vision benefits.
11 paid holidays.
Disability and basic life insurance.
Family-forming assistance.
Mental health program.
Flexible paid time off/vacation for salaried employees, plus 80 hours of paid sick time per year.
Wellness expense reimbursement.