Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - New York

Photon

4d•$40,000 - $140,000•Onsite

About The Position

As a Senior Data Engineer, you will be responsible for breaking down data silos. This role focuses on building a unified, high-performance data layer using Data Federation techniques. You won't just move data; you will architect a Data Lakehouse environment where disparate sources feel like a single, cohesive database for our analytics and AI teams.

Requirements

5+ years of experience with Python and deep expertise in Apache Spark tuning (partitioning, shuffling, caching).
Hands-on experience with Starburst Enterprise, Trino (Presto), or Dremio.
Proven track record working with Delta Lake or Iceberg architectures.
Extensive experience with AWS (EMR, S3, Glue), Azure (Databricks, ADLS), or GCP.
Expert-level SQL skills for complex analytical queries and query plan analysis.
Proficiency in designing Star/Snowflake schemas and understanding "Medallion Architecture" (Bronze, Silver, Gold layers).

Nice To Haves

Experience with Infrastructure as Code (IaC) like Terraform or Pulumi.
Familiarity with dbt (data build tool) for modeling within the federation layer.
Knowledge of Kubernetes (K8s) for deploying and scaling Spark/Trino clusters.
Background in Data Mesh or Data Fabric methodologies.

Responsibilities

Design and implement federated query layers (e.g., Starburst/Trino) to allow high-speed analytics across distributed data sources without unnecessary data movement.
Build scalable, distributed data processing pipelines using Python and Apache Spark (PySpark).
Manage and optimize modern table formats like Delta Lake, Apache Iceberg, or Hudi to bring ACID transactions to our data lake.
Optimize Spark jobs and SQL queries across the federation layer to minimize latency and manage compute costs.
Implement fine-grained access control and data masking within the federation engine to ensure data privacy across all connected platforms.