Data Engineering Lead

PURE Insurance • White Plains, AL

1d • $115,000 - $156,667 • Hybrid

About The Position

We are seeking a highly skilled and experienced hands-on Senior Data Engineer to lead the design, development, and optimization of scalable data pipelines using Databricks on Azure. This role will be instrumental in driving data architecture initiatives, enhancing data quality, and enabling advanced analytics across the enterprise.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering with at least 2 years on Databricks.
• Proficiency in Python, Scala, SQL, and Spark.
• Hands-on experience with Azure Data Services (ADF, ADLS, Synapse).
• Strong understanding of ETL, data warehousing, and data modeling concepts.
• Experience with Microstrategy, Power BI, including DAX and advanced visualizations.

Nice To Haves

• Familiarity with MLflow, LangChain, and LLM integration is a plus.
• Knowledge of Duck Creek a plus.
• Insurance Domain knowledge preferred.
• Databricks Data Engineering Professional
• Azure/AWS Data Engineering Certifications

Responsibilities

• Databricks Development & Optimization: Build and optimize distributed data processing jobs using Apache Spark on Databricks. Implement Delta Lake, DLT pipelines, dbt transformations and Medallion architecture for scalable and reliable data workflows.
• ETL & Data Integration: Design and automate ETL pipelines using Azure Data Factory, Databricks, and Synapse Analytics. Integrate data from diverse sources including Duck Creek, Intacct, Workday and external APIs.
• Data Modeling & Warehousing: Develop dimensional models (Star/Snowflake schemas), stored procedures, and views for data warehouses. Ensure efficient querying and transformation using SQL, T-SQL, and PySpark.
• Cloud & DevOps Integration: Leverage Azure DevOps, CI/CD pipelines, and GitHub for version control and deployment. Utilize Azure Logic Apps and ML Flow for workflow automation and model training.
• Security & Governance: Implement role-based access control (RBAC), data encryption, and auditing mechanisms. Ensure compliance with enterprise data governance policies.
• Collaboration & Leadership: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality data solutions. Mentor junior engineers and contribute to code reviews and architectural decisions.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level