Senior Azure Databricks Engineer
About The Position
The Senior Azure Databricks Engineer will collaborate with the Data Platforms and Integration team to architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services. The role designs and develops data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads. The engineer administers and maintains Databricks workspaces, including clusters, jobs, Unity Catalog, secrets, compute and cluster policies, and governance. The position also establishes enterprise security and data governance practices; optimizes workload performance and cost; and designs monitoring, logging, alerting, and platform health solutions. It develops reusable architecture patterns, standards, procedures, and documentation, supports migrations to Azure Databricks and Lakehouse architecture, and troubleshoots production incidents involving Databricks, Spark workloads, data pipelines, and platform infrastructure.
Requirements
- At least 8–10 years of experience designing and implementing enterprise data platforms.
- At least 5 years of hands-on Azure Databricks experience in enterprise environments.
- Experience architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
- Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
- Experience developing data engineering solutions using Apache Spark, including PySpark and/or Spark SQL.
- Experience implementing enterprise data governance using Unity Catalog, metadata management, data lineage, and fine-grained access controls.
- Experience designing and working with enterprise security architectures, including RBAC, managed identities, private endpoints, Key Vault integration, encryption, and network security.
- Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards.
- Strong understanding of dimensional modeling, Data Vault, medallion architecture, and enterprise data warehouse concepts.
- Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking.
- Strong SQL programming skills and proficiency in Python and PySpark development.
- Strong experience implementing CI/CD pipelines using Azure DevOps, automated testing, release management, and deployment automation.
Responsibilities
- Collaborate with the Data Platforms and Integration team to architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
- Design and develop enterprise data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads.
- Administer and maintain Azure Databricks workspaces, including cluster configuration, job scheduling, workspace administration, Unity Catalog, secret management, compute and cluster policies, and workspace governance.
- Define and implement enterprise security architecture, including Microsoft Entra ID integration, RBAC, PIM, encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
- Establish data governance standards using Unity Catalog, including lineage, data quality, metadata management, catalog organization, access policies, and auditing.
- Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching, partitioning, Delta optimization, Photon, and cost optimization.
- Design monitoring, logging, operational dashboards, alerting, and platform health monitoring using Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
- Develop reusable architecture patterns, reference implementations, development and coding standards, naming conventions, operational procedures, and platform documentation.
- Support migration of legacy data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
- Provide technical support and troubleshoot production incidents involving Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Career Level
Senior
Education Level
No Education Listed
