Azure Databricks Infrastructure Engineer

CLOUDSCOUTS SOFTWARE SOLUTIONS LLC

2d•Remote

About The Position

We are seeking an experienced Azure Databricks Infrastructure Engineer to execute migrations of Azure Databricks environments between Azure tenants. The successful candidate will have deep expertise in Databricks, Azure cloud services, identity and access management, networking, data engineering, and platform modernization. This role is responsible for executing a migration strategy that minimizes risk, downtime, and business disruption. The ideal candidate will combine strong technical capabilities with consulting skills and be comfortable interacting with architects, platform teams, security organizations, and business stakeholders.

Requirements

13+ years profiles only
Azure Databricks
Microsoft Entra ID
Azure Virtual Networks
Private Endpoints
Azure Key Vault
Azure Storage Accounts
Azure Data Lake Storage Gen2
Azure Monitor
Log Analytics
Azure Policy
Azure RBAC
Databricks Workspace administration
Unity Catalog
Delta Lake
Databricks Repos
MLflow
Cluster management
Cluster policies
Secret scopes
Databricks REST APIs
Terraform
Azure DevOps
GitHub Actions
PowerShell
Azure CLI
Databricks CLI
7+ years of Azure cloud experience.
5+ years of Azure Databricks administration and engineering experience.
Demonstrated experience migrating Azure resources across tenants.
Experience with large-scale data platform migrations.
Experience designing secure cloud landing zones.
Experience implementing Infrastructure as Code.
Experience working in regulated enterprise environments.

Nice To Haves

Azure Solutions Architect Expert
Databricks Administrator Certification
Databricks Solution Architect Certification
Experience with Unity Catalog migrations
Experience with mergers, acquisitions, divestitures, or tenant-to-tenant transformations

Responsibilities

Analyze tenant-specific dependencies, networking requirements, DNS, Private Endpoints, and identity integrations.
Define migration waves and cutover strategies.
Migrate Databricks workspaces between Azure tenants.
Rebuild or migrate Workspaces, Compute policies, Job schedules, Repositories, Workflows, Secret scopes, and Access controls.
Migrate notebooks, libraries, packages, and configuration artifacts.
Validate Unity Catalog and data governance configurations.
Implement infrastructure-as-code and automation where applicable.
Migrate and validate Microsoft Entra ID integrations, Managed identities, Service principals, RBAC assignments, and Databricks groups and permissions.
Coordinate Key Vault migration and secret rotation activities.
Ensure compliance with Principle of Least Privilege (PLP).
Support security reviews and governance approvals.
Assess and migrate associated datasets and storage integrations.
Validate Delta Lake tables, metadata, and schemas.
Develop rollback and recovery procedures.
Develop Infrastructure-as-Code solutions using Terraform, Bicep, or ARM Templates.
Automate migration and validation activities wherever possible.
Produce migration strategy and wave plans.
Produce detailed migration runbooks.
Produce cutover and rollback plans.
Produce operational handover documentation.