Lead Product Software Engineer - Data Systems

The Walt Disney Company • Glendale, NY

3d • Hybrid

About The Position

ESPN Product & Technology is seeking a Lead Software Engineer with deep expertise in building scalable distributed systems, platform services, and data-intensive applications to power personalized user experiences. This role is crucial for building a new real-time short-form video recommendation system, which will be the foundation of ESPN's next-generation personalization. You will collaborate with Software Engineering, Machine Learning, Product, and Platform teams to design and deliver essential systems, APIs, data platforms, and infrastructure that support real-time personalization and recommendation services at ESPN's scale.

Requirements

• 7+ years of experience building and maintaining production-grade data pipelines and distributed data processing systems
• Strong experience with modern data processing frameworks such as Spark, Flink, Beam, Kafka Streams, or equivalent.
• Experience designing and implementing real-time streaming data pipelines.
• Proficiency with SQL and schema design for large-scale analytical datasets.
• Familiarity with cloud data platforms (e.g., AWS) and modern data infrastructure components (e.g., data lakes, data warehouses, feature stores).
• Experience supporting ML workflows (model training pipelines, feature engineering, data validation).
• Strong knowledge of data quality frameworks and best practices, with hands-on experience using Databricks, Snowflake, and Apache Airflow for data pipeline orchestration and validation.
• Solid software engineering skills with experience in Python, Java, Scala, or similar languages.
• Strong problem-solving skills and ability to work independently in a fast-paced environment.
• Bachelor’s or Master’s in Computer Science, Data Engineering, or a related technical field, or equivalent practical experience.

Nice To Haves

• Prior experience building data infrastructure for personalization, recommendation systems, or other ML-powered products.
• Familiarity with ML lifecycle tools (MLflow, TFX, Kubeflow) and MLOps best practices.
• Experience implementing data validation, monitoring, and lineage tools (e.g., dbt tests, Snowflake data quality checks) to ensure high data integrity for ML models.
• Knowledge of real-time ML serving architectures and online feature generation.
• Experience optimizing large-scale data workflows for latency-sensitive applications.
• Prior experience operating in 0→1 product development or startup environments.
• Experience with tools/technologies such as Databricks, Snowflake, Kafka, AWS SQS, Kubernetes, and related cloud-native data platform components.

Responsibilities

• Design, build, and operate highly scalable software systems and services that support content discovery, personalization, and recommendation experiences.
• Develop and maintain distributed data processing platforms and service architectures that power both online and offline product workflows.
• Build foundational platform capabilities, including feature serving, model inference integration, experimentation infrastructure, and recommendation delivery services.
• Design reliable APIs and service interfaces that enable personalization capabilities across multiple ESPN products and surfaces.
• Lead architecture and technical design efforts for systems that must operate with high availability, low latency, and large-scale traffic demands.
• Partner with Machine Learning, Data Science, Product, and Platform Engineering teams to translate business objectives into scalable software solutions.
• Establish engineering standards, operational best practices, monitoring, observability, and reliability mechanisms across critical systems.
• Drive technical strategy and execution for next-generation personalization platforms and services.
• Mentor engineers and influence engineering practices across teams through technical leadership, design reviews, and architectural guidance.

Benefits

• A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior