Lead Azure Databricks Architect

EXL Talent Acquisition Team

74d

About The Position

The position involves leading the design, development, and deployment of scalable data pipelines and workflows using core Azure data services. This includes working with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS), and Azure Databricks. The role requires architecting and optimizing data solutions leveraging Medallion Architecture on Delta Lake as part of the overall Azure data platform strategy. Additionally, the candidate will develop and optimize ETL/ELT processes to ensure efficient data ingestion, transformation, and delivery.

Requirements

• Proven experience in designing and implementing data pipelines using Azure Data Factory and Azure Databricks.
• Strong understanding of Medallion Architecture and Delta Lake.
• Experience with ETL/ELT processes and data transformation techniques.
• Ability to collaborate with cross-functional teams and translate business requirements into technical solutions.
• Experience in leading and mentoring teams in a data engineering context.
• Knowledge of data governance, quality, security, and compliance standards in Azure.
• Familiarity with Azure Active Directory (AAD) and Azure Key Vault.

Nice To Haves

• Experience with Azure Synapse Analytics and Azure Data Lake Storage (ADLS).
• Knowledge of CI/CD practices and tools, particularly Azure DevOps.
• Familiarity with data science concepts and collaboration with data scientists.

Responsibilities

• Lead the design, development, and deployment of scalable data pipelines and workflows using Azure data services.
• Architect and optimize data solutions leveraging Medallion Architecture on Delta Lake using Azure Databricks.
• Develop and optimize ETL/ELT processes with Azure Data Factory and Azure Databricks.
• Collaborate with data scientists, analysts, and business stakeholders to translate business needs into Azure-based data engineering solutions.
• Lead and mentor a team of data engineers, enforcing best practices in coding, testing, deployment, and documentation.
• Ensure data governance, quality, security, and compliance standards are maintained across Azure data pipelines.
• Implement secure authentication and authorization mechanisms using Azure Active Directory (AAD).
• Manage secrets and keys securely using Azure Key Vault.
• Implement CI/CD pipelines for automated deployment and monitoring using Azure DevOps.
• Stay updated with Azure data services innovations and incorporate new technologies to enhance data infrastructure and team productivity.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Number of Employees: 5,001-10,000 employees