Data Architect - Databricks

Capgemini • Chicago, IL

3d • Hybrid

About The Position

We are seeking an accomplished Senior Data & AI Architect to lead the design and delivery of enterprise-scale data engineering and AI-driven solutions. This role requires deep expertise in Azure cloud architecture, Databricks, modern data ingestion frameworks, Building Canonical Data Models, and Medallion architecture, along with strong leadership and communication skills. The ideal candidate will define enterprise data standards, architect scalable platforms, and collaborate across global teams to deliver high-quality data products.

Requirements

• 20+ years of experience delivering Data & AI–driven projects.
• 10+ years of experience in solution architecture, primarily on cloud platforms.
• 7+ years of hands-on experience in: Data acquisition framework design
• Spark-based ingestion (CDC, batch, streaming, micro-batch)
• Handling structured and unstructured data
• Azure Data Lake Storage (ADLS Gen2) and cloud storage architectures
• Azure cloud platform and Databricks (including Unity Catalog)
• 5+ years of hands-on experience with Python, PySpark, Databricks Notebooks, GitHub, and CI/CD pipelines.
• 2+ years of experience leveraging AI-powered development tools.
• Strong understanding of enterprise data governance, security, and access management.
• Experience leading enterprise data modernization initiatives.
• Exposure to AI/ML enablement through data platforms.
• Prior experience in highly regulated or large-scale enterprise environments.

Responsibilities

• Lead architecture and solution design for large-scale Data & AI platforms on Azure.
• Design and implement data acquisition frameworks using CDC, batch, micro-batch, and real-time streaming patterns with Spark.
• Architect and manage data storage solutions using ADLS Gen2, domain/sub-domain–based storage, and data product containers.
• Build and govern Medallion (Bronze/Silver/Gold) data layers for enterprise analytics and AI use cases.
• Provide hands-on technical leadership using Python, PySpark, and Databricks Notebooks.
• Define and enforce data architecture standards, reference architectures, and design patterns.
• Design and oversee data governance, security, and access control models, including Unity Catalog.
• Integrate CI/CD pipelines, GitHub workflows, and automated deployment practices.
• Leverage AI-based code assistance tools (e.g., GitHub Copilot, Databricks Genie) to improve developer productivity and code quality.
• Collaborate effectively with onshore and offshore teams, mentoring engineers and architects.
• Communicate technical solutions clearly to business stakeholders, architects, and leadership.

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 5,001-10,000 employees