Senior Data Engineer - Databricks

Capgemini • New York, NY

2d • $89,400 - $166,398 • Onsite

About The Position

Capgemini is seeking a Senior Databricks Data Engineer to design, develop, and optimize scalable data solutions supporting analytics, machine learning, and generative AI initiatives. The role works closely with data architects, data scientists, business stakeholders, and application teams to build modern lakehouse architectures and high-performance data pipelines. The position requires onsite work at the customer location in Troy, Michigan, at least four days per week.

The engineer will build and maintain batch and real-time pipelines, enterprise ETL/ELT frameworks, Databricks Lakehouse solutions, and Delta Lake tables. The role also includes data governance and security administration with Unity Catalog, workflow and deployment management, infrastructure provisioning, CI/CD implementation, and preparing data and platform capabilities for machine learning and LLM applications.

The stated experience range is 5–8 years in data engineering. The posted base compensation range is $89,400–$166,398. Compensation may also include variable incentives, bonuses, or commissions, depending on the position and applicable laws. Capgemini provides a benefits package to regular full-time employees, with availability based on local policy and eligibility.

Requirements

• 5–8 years of experience in data engineering.
• Strong proficiency in Python (PySpark) and SQL.
• Hands-on experience with the Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Databricks Workflows, Structured Streaming, and Auto Loader.
• Experience designing and supporting large-scale enterprise data platforms.
• Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
• Experience implementing CI/CD pipelines, version control, and automated testing practices.
• Hands-on experience with Git, GitHub or GitLab, Terraform, and pytest.
• Experience with at least one major cloud platform: Azure, GCP, or AWS.
• Ability to work onsite at the customer location in Troy, Michigan at least four days per week.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
• Build ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
• Implement data quality checks, monitoring, and pipeline optimization for reliability and performance.
• Design and manage Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, and Gold layers).
• Develop and maintain Delta Lake tables and optimize query performance.
• Tune performance using Databricks best practices, including Photon Engine, Liquid Clustering, and partitioning.
• Configure and administer Unity Catalog for enterprise data governance.
• Implement catalog, schema, table, row-level, and column-level security controls.
• Enable data lineage, auditing, and compliance across the data platform.
• Develop and maintain workflows using Databricks Workflows.
• Manage deployment processes using Databricks Asset Bundles (DABs).
• Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
• Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies.
• Prepare and curate datasets for machine learning and generative AI applications.
• Support feature engineering and feature store implementations.
• Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
• Collaborate with Data Science and AI teams to operationalize machine learning workloads.

Benefits

• Paid time off, including vacation (12–25 days depending on employee grade), company-paid holidays, personal days, and sick leave, based on policy.
• Medical, dental, and vision coverage, subject to local policy and eligibility.
• Retirement savings plans, such as a 401(k) in the U.S. or RRSP in Canada, subject to local policy and eligibility.
• Life and disability insurance, subject to local policy and eligibility.
• Employee assistance programs, subject to local policy and eligibility.
• Other benefits as provided by local policy and eligibility.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed