Lead Data Engineer

Capgemini • Chicago, IL

2d

About The Position

We are seeking a highly experienced Foundry Lead Data Engineer to architect, develop, and optimize scalable data solutions using Palantir Foundry. This senior role is ideal for a data engineering expert who thrives in complex, ontology-aware environments and can lead the design of robust data pipelines and applications that drive actionable insights.
You will play a pivotal role in shaping our data strategy, collaborating with stakeholders across the organization, and ensuring the integrity, performance, and cost-efficiency of our data ecosystem. While familiarity with Generative AI (GenAI) is a plus, your core expertise should lie in data architecture, pipeline development, and Foundry platform mastery.

Requirements

• 8+ years of experience in data engineering or related roles.
• Strong proficiency in PySpark, Spark SQL, and Python.
• Deep hands-on experience with Palantir Foundry, including:
• Data Connections
• Pipeline Builder
• Contour
• Code Repositories
• Ontology Management
• Workshop and Quiver
• Solid understanding of data modeling, data governance, and access control.
• Proven ability to troubleshoot complex data issues and optimize performance.
• Excellent communication and collaboration skills.

Nice To Haves

• Experience with cloud platforms (e.g., Azure).
• Familiarity with Snowflake and its integration with Foundry.
• Exposure to Generative AI (GenAI) and its application in data workflows.
• Knowledge of data security, compliance, and metadata management.

Responsibilities

• Design, build, and maintain end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Python.
• Architect scalable data frameworks that support high-performance analytics and operational workflows.
• Conduct data cleansing, transformation, and validation to ensure data quality and consistency.
• Configure and manage data connections, Pipeline Builder, and Contour for data exploration and visualization.
• Develop and maintain ontology objects, ensuring semantic consistency and reusability across applications.
• Manage code repositories and version control within Foundry, promoting modular and maintainable engineering practices.
• Collaborate with data owners and domain experts to ingest, transform, and model data into ontology-driven applications such as Workshop and Quiver.
• Maintain clarity on the end-to-end architecture and data flow, ensuring seamless integration from ingestion to ontology.
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Engage with senior leadership to maintain a feedback loop for Foundry program improvements.
• Provide mentorship and technical guidance to junior engineers and analysts.
• Establish and maintain cost monitoring and reporting for Foundry-related operations.
• Lead efforts in troubleshooting, performance tuning, and platform stability.
• Stay current with Foundry platform updates, emerging features, and best practices.
• Identify opportunities for automation, optimization, and innovation in data workflows.
• Explore and evaluate GenAI capabilities for potential integration into data processes.

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 5,001-10,000 employees