Lead Software Engineer - Big Data Infrastructure

About the position

Disney Entertainment & ESPN Product & Technology (DEEP&T) is focused on creating magical viewing experiences and transforming Disney's media business. The Big Data Infrastructure team manages big data services such as Hadoop, Spark, Flink, and Presto, supporting large-scale data and physical resources. The role involves building and maintaining large-scale data systems and requires experience in mission-critical infrastructure.

Responsibilities

• Develop, scale, and improve in-house/cloud and open-source large-scale data processing engines or framework systems (e.g. Spark, Flink, Presto/Trino, etc).
• Investigate new big data technology, and apply it to the DisneyStreaming production environment.
• Build next-gen cloud-based big data infrastructure for batch and streaming data applications, and continuously improve performance, scalability and availability.
• Handle architectural and design considerations such as performance, scalability, reusability, and flexibility issues.
• Advocate engineering best practices, including the use of design patterns, code review, and automated unit/functional testing.
• Work together with other engineering teams to influence them on big data system design and optimization.
• Define and lead the adoption of best practices and processes. Collaborate with senior internal team members and external stakeholders to gather requirements and drive implementation.
• Collaborate efficiently with Product Managers and other developers to build datastores as a service.

Requirements

• At least 7 years of professional programming and design experience.
• Experience with big data-related components (e.g. HDFS, HBase, Yarn, Hive, Spark, Flink, Presto, Impala, Terraform, EKS, Spinnaker, IAM, EMR, etc).
• Experience in building in-house big data infrastructure.
• Experience in developing and optimizing ETL and ad-hoc query engines (e.g. Spark, Flink, Hive, Presto/Trino, GreenPlum).
• Experience in CICD, fine-tuned metrics, security and compliance enhancement on compute engines.
• Experience in latest data formats (Iceberg, Delta, Hudi).

Nice-to-haves

• Experience in catalog and metadata management.
• Experience in developing and optimizing Hadoop-related and containerized technologies (e.g. HDFS, HBase, Yarn, Kubernetes, docker, RocksDB).
• Demonstrated ability with cloud infrastructure technologies, including Terraform, K8S, IAM, ELB, Ranger, KMS, S3, Glue.
• Experience in managing a big data cluster with over 1000 nodes.

Benefits

• Medical, financial, and/or other benefits depending on the level and position offered.
• Bonus and/or long-term incentive units may be provided as part of the compensation package.