Senior Databricks Data Engineer

CGI•Miami, FL

4d•Hybrid

About The Position

CGI is seeking a hands-on Senior Databricks Data Engineer to design, develop, and implement scalable enterprise data integration solutions. The role focuses on Databricks, Apache Spark, and production-grade ETL/ELT pipelines. The engineer independently owns the full engineering lifecycle, from requirements analysis and solution design through technical review, development, testing, deployment, implementation, and production support. The position is based in Miami, Florida, in a hybrid working environment. Experience with AWS, Azure, or Google Cloud Platform is acceptable; no specific cloud provider is preferred. The role includes developing batch and incremental pipelines from files, databases, and enterprise data sources; building reusable processing solutions; optimizing workloads; and implementing data quality, reconciliation, monitoring, and error handling. It also involves creating SQL transformations and curated datasets, dimensional models, and reporting views to support analytics and downstream consumers. The engineer will implement orchestration and CI/CD practices, troubleshoot pipeline and production issues, collaborate with architects, engineering teams, integration teams, and business stakeholders, and document technical designs and operational procedures. CGI describes a culture centered on ownership, teamwork, respect, and belonging, and offers eligible professionals benefits from their first day of employment. Applications are anticipated to be accepted through November 13, 2026.

Requirements

At least 5 years of relevant experience in data engineering, data integration, or related disciplines.
Strong, recent hands-on development experience with Databricks.
Proven experience designing, building, and implementing production ETL/ELT pipelines.
Advanced hands-on proficiency in Apache Spark, PySpark, and Python.
Strong SQL skills, including complex joins, window functions, data transformations, and query performance tuning.
Experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform (GCP).
Experience with workflow orchestration tools such as Apache Airflow, Databricks Workflows, or equivalent platforms.
Experience with CI/CD practices, source control, and automated deployment of data pipelines.
Ability to independently own the full development lifecycle, including solution design, technical review, implementation, and production support.
Ability to evaluate requirements, make sound technical design decisions, and implement approved solutions.
Strong problem-solving skills and a track record of independent execution, accountability, and ownership in fast-paced environments.
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Nice To Haves

Experience calling external REST APIs from Databricks notebooks, including authentication, request handling, and ingestion of API responses; building or exposing APIs is not required.
Streaming and event-driven data processing experience.
Delta Lake and medallion architecture experience.
Unity Catalog, governance, and lineage experience.
Metadata-driven and reusable pipeline framework experience.
Experience with enterprise-scale Databricks implementations.
Experience with complex enterprise data integration environments.
Strong analytical skills, technical documentation, cross-team collaboration, and experience in Agile delivery environments.
Spark performance tuning and cost optimization experience.
Batch and incremental data processing experience.
Data quality, reconciliation, monitoring, and observability experience.
Data warehousing, fact/dimension modeling, and curated data layer experience.
Production troubleshooting and operational reliability experience.
Technical documentation and design review participation.
Cross-functional communication and Agile delivery experience.

Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, Python/PySpark, and SQL.
Build batch and incremental data integration pipelines from files, databases, and enterprise data sources.
Develop reusable and maintainable data processing solutions.
Optimize Spark workloads for performance, scalability, reliability, and cost efficiency.
Implement data quality controls, reconciliation, monitoring, and error handling.
Own the engineering lifecycle from requirements analysis and solution design through technical review, implementation, and deployment.
Prepare and present technical solution designs for architecture and engineering review, incorporate feedback, and implement approved designs.
Make technical decisions and proactively resolve engineering challenges while taking accountability for deliverables, quality, timelines, and production readiness.
Develop complex SQL transformations for cleansing, enrichment, and aggregation.
Design and maintain curated datasets, dimensional models, and reporting views to support enterprise analytics, reporting, and downstream consumers.
Implement and maintain pipeline orchestration using Airflow, Databricks Workflows, or comparable tools.
Apply CI/CD practices for version control, automated deployment, and release management.
Troubleshoot pipeline failures and production performance issues.
Collaborate with data architects, integration teams, engineering teams, and business stakeholders.
Document technical designs, implementation decisions, and operational procedures.