As the Lead Data Engineer, you will report to the Engineer Manager (Data Services) and continuously collaborate closely with key stakeholders across the business to solve critical technical challenges.
Take ownership for achieving objectives and key results for your team, oversee & own technical solutions, communicate schedule, status, and milestones
Collaborate with cross functional partners (Product Managers, Data Scientists, Machine Learning Engineers, Software Engineers, and Business teams) to build data products
Communicate effectively with both technical and non-technical stakeholders. Translate technical concepts into clear, accessible terms.
Develop, optimize, and maintain complex ETL processes for data movement and transformation
Review code and provide technical guidance to ensure adherence to high-quality standards and best practices in data engineering
Develop APIs and microservices to expose and integrate data products with software systems
Implement monitoring, logging, and alerting systems to proactively identify and resolve issues
Ensure data quality, security, and compliance with relevant regulations and standards
Stay current with industry trends, emerging technologies, and best practices in data engineering. Foster a culture of continuous learning and skill development within the team
BS, MS, or PhD in Computer Science, Mathematics, Statistics, Engineering, Operations Research, or a related quantitative field
7+ years of experience as a Data Engineer or Software Engineer developing and maintaining data pipelines, infrastructure and architecture
Strong programming skills in Python with a solid understanding of core computer science principles
Knowledge of relational and dimensional data modeling for building data products
Experience with data quality checks and data monitoring solutions
Experience orchestrating complex workflows and data pipelines using Airflow or similar tools
Proficiency with Git, CI/CD pipelines, Docker, and Kubernetes
Experience architecting solutions on AWS or equivalent public cloud platforms
Experience developing data APIs, microservices, and event-driven systems to integrate data products
Strong interpersonal and verbal communication skills
Proven leadership experience with the ability to mentor and guide a team
Familiarity with data mesh concepts
Domain knowledge in recommender systems, fraud detection, personalization, and marketing science
Understanding of vector databases, knowledge graphs, and other advanced data organization techniques
Hands-on experience with tools such as Snowflake, Postgres, DynamoDB, Kafka, Fivetran, dbt, Airflow, Docker, Kubernetes, SageMaker, Datadog, PagerDuty, data observability tools, and data governance tools