Lead Data Engineer - Experimentation Platform - 1633
About The Position
We are seeking a Lead Data Engineer to join our team and design and build scalable data platforms that support experimentation and A/B testing. This role involves developing batch and streaming data pipelines for large-scale user and product datasets, building reusable datasets and frameworks, and designing dimensional data models. You will be responsible for implementing automated data quality, validation, monitoring, lineage, and governance, as well as building production-grade deployment pipelines with CI/CD and observability. This position requires partnering with Product, Engineering, Data Science, and Analytics teams to deliver scalable data solutions and optimizing data infrastructure for experimentation, personalization, and machine learning workloads. Additionally, you will mentor engineers and establish best practices for large-scale data engineering.
Requirements
- Python
- SQL
- Data Engineering
- ETL / ELT
- Apache Spark
- Databricks
- Snowflake
- Apache Kafka
- Apache Airflow
- Streaming Data Pipelines
- Data Modeling
- Data Warehousing / Lakehouse
- A/B Testing / Experimentation Platforms
- CI/CD for Data Pipelines
- Data Quality & Data Governance
- Cloud Data Platforms
- Bachelor's degree in Computer Science, Engineering, Mathematics, Statistics, or a related technical field.
- 7+ years of experience in data engineering or large-scale data platforms.
- Strong experience with distributed data processing and cloud-based data architectures.
- Hands-on experience with Python, SQL, Spark, Databricks, Snowflake, Kafka, and Airflow.
- Strong understanding of data modeling, ETL/ELT, streaming architectures, and lakehouse concepts.
- Experience building experimentation, analytics, personalization, or ML data platforms.
- Experience implementing CI/CD, automated testing, monitoring, and data governance.
- Strong system design and architecture experience.
- Experience mentoring engineers and leading technical initiatives.
Nice To Haves
- Experimentation platforms or A/B testing infrastructure.
- Causal inference or product analytics experience.
- ML feature engineering and model lifecycle pipelines.
- Infrastructure automation and observability.
- Subscription, streaming media, advertising, or consumer product experience.
- MS or PhD in a related technical field.
Responsibilities
- Design and build scalable data platforms supporting experimentation and A/B testing.
- Develop batch and streaming data pipelines for large-scale user and product datasets.
- Build reusable datasets and frameworks for experimentation, analytics, and product measurement.
- Design dimensional data models and analytics-ready data products.
- Implement automated data quality, validation, monitoring, lineage, and governance.
- Build production-grade deployment pipelines with CI/CD and observability.
- Partner with Product, Engineering, Data Science, and Analytics teams to deliver scalable data solutions.
- Optimize data infrastructure supporting experimentation, personalization, and machine learning workloads.
- Mentor engineers and establish best practices for large-scale data engineering.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
