Azure Databricks Architect

Opinov8 • St. Louis, MO

11d • Remote

About The Position

We are looking for an experienced Azure Databricks Architect to design, implement, and govern enterprise-scale data platforms on Microsoft Azure. The ideal candidate will possess deep expertise in Azure Databricks, Lakehouse architecture, cloud-native data solutions, and DevOps best practices. This role goes beyond data engineering and requires strong capabilities in platform architecture, CI/CD automation, cloud governance, security, cost optimization, and operational excellence. The successful candidate will work closely with business stakeholders, data engineers, architects, and DevOps teams to build scalable, secure, and highly available analytics platforms.

Requirements

• 7+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Architecture, or Data Platform Consulting
• 3+ years of hands-on experience architecting solutions with Azure Databricks
• Strong expertise in Apache Spark, PySpark, Delta Lake, and Databricks Lakehouse architecture
• Hands-on experience with Unity Catalog, data governance, lineage, permissions models, and catalog isolation strategies
• Experience designing and managing Databricks Metastore and multi-workspace architectures
• Strong knowledge of Databricks administration, user provisioning, RBAC, workspace management, and security controls
• Extensive experience designing cloud-native solutions on Microsoft Azure
• Strong knowledge of Azure Data Factory, Azure Storage, Azure Key Vault, Azure Monitor, Azure Entra ID, and Azure networking concepts
• Experience designing streaming and low-latency data pipelines using Databricks Structured Streaming
• Hands-on experience with CDC, Change Data Feed (CDF), and modern data ingestion patterns
• Hands-on experience building and maintaining CI/CD pipelines using Azure DevOps and/or GitHub Actions
• Experience implementing Infrastructure as Code using Terraform, ARM Templates, or Bicep
• Strong understanding of DevOps principles, deployment automation, and environment management
• Strong SQL and Python programming skills
• Excellent communication, stakeholder management, and consulting skills

Nice To Haves

• Knowledge of Terraform or Infrastructure as Code (IaC)
• Databricks certifications
• Microsoft Azure certifications
• Experience with Lakeflow Connect and managed connectors
• Experience with Kubernetes and containerized workloads
• Experience with Microsoft Fabric

Responsibilities

• Architect and design enterprise-scale data platforms leveraging Azure Databricks and Azure cloud services
• Define and implement Lakehouse architecture patterns using Delta Lake and Databricks best practices
• Design scalable data ingestion, transformation, and serving frameworks supporting both batch and streaming workloads
• Design and implement real-time and near real-time data processing solutions using Structured Streaming, CDC, Change Data Feed (CDF), and event-driven architectures
• Design and manage multi-workspace Databricks environments and shared Metastore architectures
• Establish platform governance, security controls, cluster policies, RBAC models, and access management frameworks
• Design and implement CI/CD pipelines for Databricks assets, infrastructure, and data workloads
• Automate infrastructure provisioning using Infrastructure as Code (Terraform, ARM, or Bicep)
• Develop deployment strategies and release management processes across development, staging, and production environments
• Optimize Databricks workspaces, clusters, jobs, and storage configurations for performance and cost efficiency
• Implement monitoring, alerting, logging, and observability solutions for platform operations
• Define and enforce data architecture standards, data quality frameworks, and governance policies
• Integrate Azure Databricks with Azure Data Factory, Azure Synapse, Azure Storage, Azure Key Vault, Azure Entra ID, and other Azure services
• Support Lakeflow Connect, managed connectors, CDC implementations, and enterprise data ecosystem interoperability
• Collaborate with engineering and business teams to translate requirements into scalable technical solutions
• Provide architectural guidance, technical leadership, and mentorship to engineering teams

Benefits

• Digital-First Approach: Great talent knows no borders! You can work from wherever you are — we hire and collaborate with professionals worldwide.
• Remote Work Model: Balance your professional and personal life with our flexible working conditions, empowering you to deliver your best from anywhere.
• Exciting Projects: Dive into impactful projects across industries that challenge and spark creativity.
• Boost Your Expertise: Grow your career with continuous learning, development opportunities, and hands-on experience.
• Join the Best Team Ever: Collaborate with our diverse and cross-cultural team of passionate technologists and creative thinkers.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed