Lead Databricks Engineer
About The Position
Capgemini is seeking a Lead Data Engineer to design, develop, and optimize scalable data ingestion and processing solutions using Databricks native capabilities. The role requires deep data engineering, large-scale data integration, and cloud-based data platform expertise, including hands-on experience with Databricks Native Data Loader, Lakeflow Connect, Delta Lake, and enterprise data migration from relational databases such as Oracle, MySQL, and SQL Server. The position leads the design and implementation of modern data pipelines, mentors engineering teams, and works with business and technology stakeholders to deliver reliable, scalable, high-performing data solutions. It is located in Los Angeles, California, with a hybrid work arrangement.
Requirements
- Deep expertise in data engineering, large-scale data integration, and cloud-based data platforms.
- Hands-on experience with Databricks Native Data Loader, Lakeflow Connect, and Delta Lake.
- Experience with enterprise data migration from relational databases such as Oracle, MySQL, and SQL Server.
- Experience designing and implementing scalable data ingestion and processing solutions using Databricks native capabilities.
- Knowledge of batch and streaming pipelines, APIs, Databricks, Spark, and cloud-native technologies.
- Ability to provide technical leadership and mentor engineers.
- Ability to collaborate with business and technology stakeholders.
Responsibilities
- Design and implement enterprise-scale data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities.
- Build and optimize batch and streaming pipelines that ingest data from Oracle, MySQL, SQL Server, APIs, and other enterprise sources.
- Develop ELT/ETL pipelines using Databricks, Spark, Delta Lake, and cloud-native technologies.
- Implement end-to-end data movement, transformation, validation, and monitoring solutions.
- Lead architectural decisions regarding data ingestion, storage, processing, and governance.
- Create and maintain effective data pipeline architectures and build scalable pipelines that transform raw and unstructured data into analytics-ready datasets.
- Assemble large, complex datasets that meet functional and non-functional business requirements.
- Identify and implement process improvements, including automation, infrastructure optimization, and scalability enhancements.
- Develop infrastructure to extract, transform, and deliver data from diverse sources using distributed processing technologies.
- Build and maintain integrations for data extraction, transformation, and loading.
- Implement data validation and quality monitoring processes; investigate data issues and recommend corrective actions.
- Write unit and integration tests, use Test-Driven Development practices, and maintain technical documentation.
- Ensure platform reliability, optimize performance, and follow enterprise standards.
- Provide technical leadership and mentorship to junior engineers.
- Collaborate with Product, Engineering, Analytics, Architecture, and Business stakeholders to understand requirements and deliver solutions.
- Promote best practices in coding, DevOps, CI/CD, testing, and documentation.
- Participate in architecture reviews, code reviews, and technical design discussions.
- Support Agile development, including sprint planning and estimation.
Benefits
- Paid time off, including vacation of 12–25 days depending on employee grade, company-paid holidays, personal days, and sick leave.
- Medical, dental, and vision coverage (or provincial healthcare coordination in Canada).
- Retirement savings plans, such as a 401(k) in the U.S. or RRSP in Canada.
- Life and disability insurance.
- Employee assistance programs.
- Other benefits as provided by local policy and eligibility.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
