Sr Data Engineer

The Walt Disney Company • Burbank, CA

2d • Onsite

About The Position

Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more, all working to build and advance the technological backbone for Disney’s media business globally. The team marries technology with creativity to build world-class products, enhance storytelling, and drive velocity, innovation, and scalability for our businesses. We are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers. We work with every part of The Walt Disney Company’s media portfolio to advance the technological foundation and consumer media touch points serving millions of people around the world. Our Technologists are designing and building the products and platforms that will power our media, advertising, and distribution businesses for years to come. More than ever, Disney’s technology and products serve as a signature doorway for fans' connections with the company’s brands and stories. Disney+. Hulu. ESPN. ABC. ABC News…and many more. These products and brands – and the unmatched stories, storytellers, and events they carry – matter to millions of people globally. We develop and implement groundbreaking products and techniques that shape industry norms, and solve complex and distinctive technical problems. Commerce, Data & Identity provides the core product management functions for areas crucial to Disney’s media businesses. These include initiatives and products that power digital commerce, identity, and growth, as well as those that reach uniquely across The Walt Disney Company enterprise, such as messaging and privacy, among others. Additionally, it is responsible for the data engineering, science, and products for Disney Entertainment & ESPN, along with their interconnection with other parts of The Walt Disney Company. The Media and Session Data Product (MSDP) team for the Data organization within the DEEPT organization is in search of a Senior Software Engineer. As a member of the MSDP team you will establish set of data pipelines and datasets which are a vital key to success – enabling dozens of engineering and analytical teams to unlock the power of media session data to drive key business decisions and provide engineering, analytics, and operational teams the critical information necessary to scale the largest streaming service. Expanding, scaling, and standardizing the core foundational principles through consistent observability, lineage, data quality, logging, and alerting across all engineering teams in the Data organization is imperative to the creation of a single pane of glass. The MSDP team is seeking to grow their team of world class Software Engineers that share their charisma and enthusiasm for making a positive impact!

Requirements

• 5+ years of hands-on data engineering experience.
• Proficient in Spark Structured Stream and Databricks platform (Unity Catalog, Delta Lake, Workflows, Cluster management).
• Strong programming skills in Scala (must) and Python/PySpark (must).
• Advanced SQL skills with experience in writing complex, optimized queries.
• Hands-on experience building and scheduling workflows in Apache Airflow.
• Working knowledge of AWS services for data engineering.
• Solid understanding of data modeling (star schema, slowly changing dimensions, data vault, etc.).
• Experience with CI/CD for data pipelines (Git, Jenkins, GitHub Actions, etc.).
• Excellent problem-solving, communication, and leadership skills.
• BA/BS required

Nice To Haves

• Work with Snowflake for data warehousing, cost optimization, and modern data sharing use cases.
• Familiarity with Snowflake is a strong plus.
• Experience with streaming technologies beyond Spark (Kinesis, Flink) is a plus.

Responsibilities

• Design, develop, and optimize large-scale batch and real-time data pipelines using Spark Structured Streaming on Databricks.
• Write production-grade, maintainable code primarily in Scala and PySpark (Scala preferred).
• Implement complex data transformations using Spark SQL and core Spark APIs.
• Own the end-to-end lifecycle of data products from ingestion, transformation, orchestration, to consumption by analytics, ML, and reporting teams.
• Design and maintain workflow orchestration using Apache Airflow.
• Build and manage infrastructure on AWS with some expertise in cloud-native data architectures.
• Collaborate with Data Scientists, Analysts, and Software Engineers to productionize machine learning models and analytical dashboards.
• Implement data quality, monitoring, alerting, and lineage solutions.
• Performance tuning of Spark jobs, cluster optimization on Databricks, and cost optimization on AWS.
• Lead technical design discussions, code reviews, and mentor mid/junior engineers.

Benefits

• A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Number of Employees: 5,001-10,000 employees