Data Engineer II

The Walt Disney Company • New York, NY

17h • Onsite

About The Position

Disney Entertainment and ESPN Product & Technology is seeking a passionate Data Engineer to join the Product Performance Team within the Data Organization. This team is responsible for developing reliable data solutions that measure the product performance of Disney+. The role involves designing, building, and testing core algorithms and data models for browse, playback attribution, and search data. This data is crucial for measuring Disney+ product engagement and informing business decisions. The ideal candidate is excited by solving complex business questions using data.

Requirements

• 3+ years of hands-on data engineering experience.
• Proficient in Spark Structured Stream and Databricks platform (Unity Catalog, Delta Lake, Workflows, Cluster management).
• Strong programming skills in Scala (must have) and Python.
• Advanced SQL skills with experience in writing complex, optimized queries.
• Hands-on experience building and scheduling workflows in Apache Airflow.
• Working knowledge of AWS services for data engineering.
• Solid understanding of data modeling concepts.
• Experience with CI/CD for data pipelines (Git, Jenkins, GitHub Actions, etc.).
• Bachelor’s degree in Computer Science or equivalent study required.
• Excellent problem-solving, communication, and leadership skills.

Nice To Haves

• Familiarity with Snowflake is a strong plus.
• Experience with streaming technologies beyond Spark (Kinesis, Flink) is a plus.

Responsibilities

• Design, develop, and optimize large-scale batch and real-time data pipelines using Spark Structured Streaming on Databricks.
• Work with Data Analysts to develop reports and dashboards on snowflake.
• Write production-grade, maintainable code primarily in Scala and SQL.
• Implement complex data transformations using Spark SQL and core Spark APIs.
• Own the end-to-end lifecycle of data products from ingestion, transformation, orchestration, to consumption by analytics, ML, and reporting teams.
• Design and maintain workflow orchestration using Apache Airflow or Databricks.
• Build and manage infrastructure on AWS with some expertise in cloud-native data architectures.
• Implement data quality, monitoring, alerting, and lineage solutions.
• Performance tuning of Spark jobs, cluster optimization on Databricks, and cost optimization on AWS.

Benefits

• A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level