Senior Azure Databricks Engineer

Cynet Systems•Glendale, AZ

About The Position

Collaborate with the Data Platforms and Integration team to architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services. Design and develop data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads. Administer and maintain Databricks workspaces, governance, security, monitoring, and operational capabilities; optimize workloads for performance and cost; and support migrations to modern Lakehouse architecture. Provide technical support and troubleshoot production incidents involving Databricks, Spark workloads, data pipelines, and platform infrastructure.

Requirements

At least 8–10 years of experience designing and implementing enterprise data platforms.
At least 5 years of hands-on Azure Databricks experience in enterprise environments.
Experience architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
Experience administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
Experience developing data engineering solutions using Apache Spark, PySpark, and/or Spark SQL.
Experience implementing enterprise data governance using Unity Catalog, metadata management, data lineage, and fine-grained access controls.
Experience designing and working with enterprise security architectures, including RBAC, managed identities, private endpoints, Key Vault integration, encryption, and network security.
Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards.
Strong understanding of data modeling techniques, including dimensional modeling, Data Vault, medallion architecture, and enterprise data warehouse concepts.
Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking.
Strong SQL programming skills and proficiency in Python and PySpark development.
Strong experience implementing CI/CD pipelines using Azure DevOps, automated testing, release management, and deployment automation.

Responsibilities

Architect and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
Design and develop enterprise data engineering frameworks for batch, streaming, ELT/ETL, and real-time analytics workloads.
Administer and maintain Azure Databricks workspaces, including cluster configuration, job scheduling, workspace administration, Unity Catalog, secret management, compute and cluster policies, and workspace governance.
Define and implement enterprise security architecture, including Microsoft Entra ID integration, RBAC, PIM, encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
Establish data governance standards using Unity Catalog, including data lineage, data quality, metadata management, catalog organization, access policies, and auditing.
Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching, partitioning, Delta optimization, Photon, and cost optimization practices.
Design monitoring, logging, operational dashboards, alerting, and platform health monitoring using Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
Develop reusable architecture patterns, reference implementations, development standards, naming and coding conventions, operational procedures, and platform documentation.
Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
Provide technical support and troubleshoot production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.