Azure ADF Lead

Cloudious•Houston, TX

About The Position

Seeking an Associate Principal with 11 to 15 years of experience in designing, developing, and managing scalable data integration solutions using Azure Data Factory (ADF). This role involves leading and managing Azure Data Factory environments, designing highly available and fault-tolerant data integration solutions within the Microsoft Azure ecosystem, and collaborating with cross-functional teams to support cloud adoption and modernization of data applications. The position also includes participating in Agile development processes, contributing to CI/CD pipelines, developing infrastructure as code, monitoring pipeline performance, and providing technical mentorship to junior team members. The role requires participation in on-call rotations to address production issues.

Requirements

11 to 15 years of experience in designing, developing, and managing scalable data integration solutions using Azure Data Factory (ADF).
Azure Data Factory

Responsibilities

Design, develop, and implement robust and scalable data pipelines and workflows using Azure Data Factory to support enterprisewide data integration needs.
Collaborate with data engineers, architects, and business stakeholders to gather and translate complex data requirements into efficient ADF solutions.
Develop and maintain ETL/ELT processes, ensuring high data quality, reliability, and optimal performance.
Optimize ADF pipelines for performance, scalability, and cost efficiency.
Integrate Azure Data Factory with other Azure services such as Azure Data Lake, Azure SQL Database, and Azure Databricks to build comprehensive cloud data solutions.
Ensure compliance with data governance, security policies, and best practices for cloud-based data environments.
Document technical specifications, design decisions, and development processes related to data pipelines.
Stay current with the latest trends and best practices in Azure data services and cloud data engineering.
Lead and manage Azure Data Factory environments, including provisioning, configuration, monitoring, and administration.
Design and implement highly available, fault-tolerant, and resilient data integration solutions within the Microsoft Azure ecosystem.
Analyze and improve existing data workflows, operational standards, and automation opportunities.
Collaborate with cross-functional teams to support cloud adoption and modernization of data applications.
Participate in Agile development processes and contribute to continuous integration and continuous deployment (CI/CD) pipelines for data integration solutions.
Develop infrastructure as code (IaC) using Azure native tools to automate deployment of data integration platforms.
Monitor pipeline performance and implement data protection best practices to prevent data loss.
Provide technical mentorship, guidance, and leadership to junior team members and peers.
Participate in on-call rotations to promptly address and resolve critical production issues.