Azure Data Lead

Cynet Systems • New York City, NY

16h

About The Position

We are seeking an experienced Azure Data Lead to spearhead large-scale Databricks modernization programs. This role involves analyzing existing Python applications, redesigning single-node processing logic for distributed Spark execution, and building robust enterprise-scale data pipelines on Azure. You will be instrumental in implementing Delta Lake solutions, optimizing Spark jobs, and ensuring data quality and reliability.

Requirements

• Expertise in Python, OOP, and advanced Python design patterns.
• Proficiency in PySpark, Spark SQL, and SQL.
• Experience with Azure Databricks, Azure Data Factory, and ADLS Gen2.
• Knowledge of Apache Spark, Delta Lake, Data Lakehouse architecture, and distributed computing.
• Experience with MS SQL and Oracle PL/SQL.
• Proven experience as a senior data engineering leader with delivery of large-scale Databricks modernization programs.
• Strong verbal and written communication skills.

Nice To Haves

• Certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python.

Responsibilities

• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks.
• Build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze–Silver–Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs including partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.
• Benchmark converted applications against original Python implementations.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed