Lead Databricks Engineer

Capgemini • San Francisco, CA

2d • $99,712 - $168,716 • Hybrid

About The Position

Capgemini is seeking a highly skilled Lead Data Engineer to design, develop, and optimize scalable data ingestion and processing solutions using Databricks native capabilities. The role focuses on data engineering, large-scale data integration, and cloud-based data platforms, including hands-on work with Databricks Native Data Loader, Lakeflow Connect, Delta Lake, and enterprise data migration from relational databases such as Oracle, MySQL, and SQL Server. The Lead Data Engineer will guide the design and implementation of modern data pipelines, mentor engineering teams, and work with business and technology stakeholders to deliver reliable, scalable, high-performing data solutions.

The position is based in Los Angeles, California, and is described as hybrid. The posted base compensation range is $99,712–$168,716. The role may also be eligible for variable incentives, bonuses, or commissions. Capgemini offers eligible regular, full-time employees a benefits package that may include paid time off, company holidays, personal days, sick leave, medical, dental and vision coverage, retirement savings plans, life and disability insurance, and employee assistance programs.

Requirements

• Hands-on experience with Databricks Native Data Loader, Lakeflow Connect, and Delta Lake.
• Expertise in data engineering, large-scale data integration, and cloud-based data platforms.
• Experience with enterprise data migration from relational databases such as Oracle, MySQL, and SQL Server.
• Experience building batch and streaming data pipelines and ELT/ETL solutions.
• Experience with Databricks, Spark, and cloud-native technologies.
• Ability to provide technical leadership and mentor junior engineers.

Responsibilities

• Design and implement enterprise-scale data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities.
• Build and optimize batch and streaming pipelines that ingest data from Oracle, MySQL, SQL Server, APIs, and other enterprise sources.
• Develop ELT/ETL pipelines using Databricks, Spark, Delta Lake, and cloud-native technologies.
• Implement end-to-end data movement, transformation, validation, and monitoring solutions.
• Lead architecture decisions concerning data ingestion, storage, processing, and governance.
• Create and maintain optimal data pipeline architecture and build scalable pipelines that transform raw and unstructured data into analytics-ready datasets.
• Assemble large, complex datasets that meet functional and non-functional business requirements.
• Identify, design, and implement process improvements, including automation, infrastructure optimization, and scalability enhancements.
• Develop infrastructure to extract, transform, and deliver data from diverse sources using distributed processing technologies.
• Build and maintain integrations for data extraction, transformation, and loading.
• Implement data validation and monitoring processes to ensure production data is accurate and reliable.
• Perform root cause analysis on data issues and recommend corrective actions.
• Write unit and integration tests, adopt Test-Driven Development practices, and maintain technical documentation.
• Ensure platform reliability, optimize performance, and adhere to enterprise standards.
• Provide technical leadership and mentoring to junior engineers.
• Collaborate with Product, Engineering, Analytics, Architecture, and Business stakeholders to understand requirements and deliver data solutions.
• Drive best practices in coding, DevOps, CI/CD, testing, and documentation.
• Participate in architecture reviews, code reviews, and technical design discussions.
• Support Agile development processes and contribute to sprint planning and estimation.

Benefits

• Paid time off, including vacation of 12–25 days depending on employee grade, company-paid holidays, personal days, and sick leave.
• Medical, dental, and vision coverage, subject to local policy and eligibility.
• Retirement savings plans, such as a 401(k) in the U.S. or RRSP in Canada, subject to local policy and eligibility.
• Life and disability insurance.
• Employee assistance programs.
• Potential eligibility for variable incentives, bonuses, or commissions.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed