Data Engineer III | 1177

CTI Consulting • Quincy, MA

7d

About The Position

CTI Staffing is partnering with a leading retail organization to hire a Senior Databricks Data Engineer for its enterprise data team in the Greater Boston, Massachusetts area. The team is building a modern Azure data platform that integrates data from multiple retail systems. The role focuses on creating reliable incremental pipelines that help the business trust its data and act on it faster.

The engineer will design, build, and optimize ETL/ELT pipelines with Databricks and PySpark; develop change data capture (CDC) pipelines for incremental and near-real-time processing; and build scalable Azure data solutions using Delta Lake and Kafka streaming. The role also involves improving existing pipelines, ingesting data from varied sources, implementing data quality and quarantine practices, applying SCD Type 1 and Type 2 modeling, and automating workflows with Databricks Workflows and GitHub-based CI/CD while collaborating in an Agile team.

Requirements

• Extensive hands-on Databricks experience and ability to contribute from day one.
• Proven experience building end-to-end data pipelines with CDC.
• Strong PySpark and SQL skills.
• Experience working in Microsoft Azure.
• Working knowledge of Delta Lake and distributed data processing.
• Experience with Kafka or comparable streaming technologies.
• Hands-on experience with SCD Type 1 and Type 2 data modeling.
• Familiarity with GitHub and CI/CD for version control and deployment.

Nice To Haves

• Python experience beyond PySpark.
• Experience with Databricks Workflows and Unity Catalog.
• Experience with message queues and API-based ingestion.
• Retail, CPG, or supply chain data background.
• Databricks certification.

Responsibilities

• Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
• Build change data capture pipelines for incremental and near-real-time processing.
• Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
• Learn existing codebases and deliver targeted pipeline upgrades.
• Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
• Implement data quality rules and quarantine error records.
• Apply SCD Type 1 and Type 2 data modeling techniques.
• Automate workflows with Databricks Workflows and GitHub-based CI/CD.
• Work with developers, architects, and product owners in an Agile team.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed