Azure Databricks Engineer

CGI•Lafayette, LA

19h•$80,600 - $141,100•Hybrid

About The Position

The Azure Databricks Engineer will design, build, and optimize scalable data pipelines and lakehouse solutions using Azure Databricks. This role is critical for enabling advanced analytics and supporting machine learning initiatives. This role requires US Citizenship. The preferred location for this role is Lafayette, LA, however this position can also be filled out of Belton, TX, Lebanon, VA or Knoxville TN. (Hybrid role)

Requirements

BA/BS/MS in Computer Science, Engineering, or Data Science.
5+ years as a Data Engineer, with 3+ years in Azure Databricks.
Expertise in Apache Spark, Delta Lake, Python, SQL, Scala.
Proficiency in Azure Data Factory, Data Lake Storage, Synapse Analytics.
Strong understanding of ETL/ELT, data modeling, and big data technologies.

Nice To Haves

Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer.
Experience with Agile methodology, data governance, and security.

Responsibilities

Develop and maintain ETL/ELT pipelines using Databricks workflows, Delta Lake, and Apache Spark.
Architect and optimize data lakehouse solutions for efficient storage and retrieval.
Configure and manage Databricks workspaces, clusters, and implement security and cost optimization strategies.
Build real-time streaming and batch processing solutions for diverse data sources.
Collaborate with data scientists to deploy ML models and automate ML pipelines using MLflow.
Monitor pipeline performance, troubleshoot issues, and optimize Spark jobs for efficiency.