Azure Data Engineer

Amerihealth Caritas•Edgmont Township, PA

2d•Hybrid

About The Position

The Azure Data Engineer designs, builds, and optimizes large scale cloud data solutions on Microsoft Azure. The role requires deep hands on expertise with modern ingestion frameworks to ingest diverse data sources, on distributed compute, real-time streaming, and dimensional modeling, combined with agile delivery practices.

Requirements

Bachelor’s degree in computer science, Information Systems, Software Engineering
Hands on experience with Azure Data Factory (4+ years)
Azure Databricks (4+ years)
HVR (2+ years)
Kafka (4+ years)
Event Hubs (2+ years)
Snowflake (4+ years)
Azure Data Lake Storage
PySpark (8+ years)
SQL (8+ years)
Expertise in dimensional modeling, SCD implementation, and data warehouse best practices
Proven experience delivering data projects in Agile (Scrum or Kanban) teams
Practical experience with Azure DevOps for CI/CD, branching strategies, and pipeline automation
Demonstrated ability to profile and optimize Databricks jobs and large scale data processing workloads
Track record of conducting code reviews, providing technical guidance
Clear communicator able to present technical tradeoffs to technical and non technical stakeholders
Analytical approach to diagnose performance and data quality issues
Strong team player who partners effectively across product, analytics, and platform teams
Patient, constructive coach who elevates code quality and engineering practices
Comfortable with shifting priorities and iterative delivery in fast moving environments

Nice To Haves

Experience with Collibra Data governance features
Familiarity with MLOps patterns and integrating data platforms with ML pipelines
Azure certifications such as DP 700, AZ 400, or Databricks Data Engineer certification
Prior work with cost optimization and observability tools for cloud data platforms
Practical experience applying Gen AI (LLMs, embeddings, retrieval augmentation) to data engineering workflows

Responsibilities

Design and implement scalable engineering solutions using Azure Data Factory, Azure Databricks, Azure Event Hubs, Kafka, HVR, Snowflake, and Azure Data Lake Storage
Build robust ETL/ELT pipelines for batch and streaming workloads using PySpark and SQL, ensuring fault tolerance, schema evolution handling, and idempotency
Design and operate event driven ingestion and processing using Event Hubs and Kafka, and integrate CDC solutions using HVR
Tune Databricks jobs and clusters (cluster sizing, autoscaling, caching, partitioning, shuffle optimization, Delta Lake optimizations) to meet SLAs and cost targets
Use Azure DevOps for backlog management, Git workflows, YAML pipelines, automated testing, and repeatable deployments across environments
Perform code reviews, enforce coding standards and best practices, and mentor junior developers to raise team capability
Work with product owners, data scientists, analysts, and architects to translate business requirements into prioritized user stories and deliverables in Agile sprints
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Identify and tackle issues concerning data management to improve data quality
Understand and implement best practices in management of data, including master data, reference data, metadata, data quality and lineage