Senior Azure Data Engineer

Impetus Technologies • New York City, NY

7h

About The Position

The Senior Azure Data Engineer works with client and offshore teams to design, build, and support scalable data pipelines using Azure Databricks and Apache Spark. The role develops ingestion pipelines and data transformation scripts and notebooks using Spark with Python, SQL, or Scala, and implements ELT/ETL workflows for analytics and reporting. It also covers Databricks cluster configuration and optimization, data quality and validation, CI/CD automation, monitoring and troubleshooting, and platform cost management. Working knowledge of dbt is a plus.

Requirements

• 5 to 8 years of experience.
• Experience with Azure Databricks and Apache Spark.
• Ability to develop Spark data transformation scripts and notebooks using Python, SQL, or Scala.
• Experience implementing ELT/ETL workflows.
• Experience configuring and optimizing Databricks clusters and managing cluster policies.
• Experience with data quality and validation, CI/CD automation, monitoring, troubleshooting, and cost optimization.

Nice To Haves

• Working knowledge of dbt.

Responsibilities

• Work with client and offshore teams.
• Develop scalable and efficient data ingestion pipelines using Azure Databricks and Apache Spark.
• Create and maintain data transformation scripts and notebooks using Spark with Python, SQL, or Scala.
• Implement ELT/ETL workflows to prepare data for analytics and reporting.
• Configure Azure Databricks clusters for performance, cost-efficiency, and resource utilization.
• Manage cluster policies, including autoscaling, instance types, and runtime versions.
• Monitor cluster health and tune Spark configurations to improve job execution times.
• Implement data quality checks and validation processes within Databricks notebooks and workflows.
• Develop and maintain CI/CD pipelines to automate deployment of Databricks notebooks, jobs, and configurations.
• Monitor Databricks jobs, clusters, and pipelines for failures or performance bottlenecks, including platform usage metrics.
• Troubleshoot and resolve issues involving data processing, cluster performance, job execution, and the platform.
• Track and analyze Databricks usage and costs; optimize cluster configurations and usage patterns to reduce expenses.
• Implement policies to control resource consumption and prevent cost overruns.
• Provide technical support and guidance on Databricks platform usage and best practices.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed