Staff Software Engineer, Data Infrastructure

Zocdoc•New York, NY

7d•Hybrid

About The Position

As a Staff Software Engineer (Data Infrastructure), you’ll lead the design and development of the software that underpins our data platform: ingestion frameworks, execution services, orchestration, metadata, data governance, and developer experience. Your focus is building APIs, libraries, and services that make data producers/consumers effective, while optimizing reliability, performance, and spend on AWS. You’ll enjoy this role if you are… A builder of platforms and frameworks (not just point pipelines). Comfortable with distributed systems abstractions (compute scheduling, storage layout, back‑pressure, retries, idempotency). Excited by lakehouse tech and modern data contracts, and you want to create self‑service for hundreds of use cases. Your day to day is… Design and ship platform services for ingestion, transformation, orchestration, and metadata (e.g., service‑backed interfaces for Dagster/Airflow, lineage, quality, and data contracts). Build execution & scheduling capabilities for Spark/SQL jobs (queuing, prioritization, retries, resource isolation on EMR/EKS/Databricks), focusing on throughput and developer experience. Implement lakehouse features (Delta/Iceberg): schema evolution, partitioning, compaction, vacuum, snapshotting, ACID guarantees, and table‑format governance Optimize Snowflake and other warehouses: cost controls, query profiling/pruning, workload isolation, RBAC; expose safe self‑service patterns. Deliver SDKs, CLIs, and templates that standardize how teams build reliable data products; enable CI/CD for data and contract testing. Work across AWS (S3, EMR/EKS, Glue/Athena, Lambda, Kinesis/MSK) with IaC (Terraform) and strong observability (Datadog/CloudWatch).

Requirements

8+ years building backend/platform software with Python/Scala/Java and strong SQL; proven track record designing distributed systems.
Deep experience with Spark (Databricks or EMR/EKS) and AWS data services; solid grasp of scheduler/executor behavior and performance tradeoffs.
Hands‑on data warehouse optimization (Snowflake ideal; others welcome).
Experience building platform APIs/SDKs that other engineers adopt; excellent collaboration and technical leadership.

Nice To Haves

Experience at petabyte‑scale data platforms, distributed big data compute, or lakehouse engines (Delta/Iceberg).
Familiarity with metadata/governance tech (Unity Catalog, Collibra, Lake Formation).

Responsibilities

Design and ship platform services for ingestion, transformation, orchestration, and metadata (e.g., service‑backed interfaces for Dagster/Airflow, lineage, quality, and data contracts).
Build execution & scheduling capabilities for Spark/SQL jobs (queuing, prioritization, retries, resource isolation on EMR/EKS/Databricks), focusing on throughput and developer experience.
Implement lakehouse features (Delta/Iceberg): schema evolution, partitioning, compaction, vacuum, snapshotting, ACID guarantees, and table‑format governance
Optimize Snowflake and other warehouses: cost controls, query profiling/pruning, workload isolation, RBAC; expose safe self‑service patterns.
Deliver SDKs, CLIs, and templates that standardize how teams build reliable data products; enable CI/CD for data and contract testing.
Work across AWS (S3, EMR/EKS, Glue/Athena, Lambda, Kinesis/MSK) with IaC (Terraform) and strong observability (Datadog/CloudWatch).

Benefits

Flexible, hybrid work environment at our convenient Soho location
Unlimited Vacation
100% paid employee health benefit options (including medical, dental, and vision)
Commuter Benefits
401(k) with employer funded match
Corporate wellness program with Wellhub
Sabbatical leave (for employees with 5+ years of service)
Competitive paid parental leave and fertility/family planning reimbursement
Cell phone reimbursement
Catered lunch everyday along with beverages and snacks
Employee Resource Groups and ZocClubs to promote shared community and belonging
Great Place to Work Certified