Databricks Engineer

SBA Communications•Boca Raton, FL

2d•Onsite

About The Position

The Databricks Engineer is a hands-on individual contributor role within the Database Engineering team at SBA Communications, responsible for provisioning, administering, and governing the enterprise Databricks platform on Microsoft Azure. Reporting to the Manager, Database Engineering, this position requires a technically skilled professional with deep expertise in Databricks workspace administration, Unity Catalog, identity and access management, infrastructure as code, and Azure data platform integration. The Databricks Engineer will work closely with the Cloud Engineering team to ensure infrastructure-level alignment across Azure networking, storage, and security, and will serve as the primary infrastructure lead. The role plays a critical part in enforcing data governance standards, maintaining the Medallion architecture, supporting EDW/ODS platform operations, and ensuring the Databricks environment meets organizational security, compliance, and cost management requirements.

Requirements

Bachelor's Degree in Computer Science, MIS or equivalent; and 3-5 years of progressive experience in data platform or cloud infrastructure engineering, with hands-on expertise in Databricks or Microsoft Azure and enterprise data governance practices.
Databricks Certified Data Engineer preferred.
Microsoft Certified: Azure Administrator Associate (AZ-104) preferred.
Microsoft Certified: Azure Data Engineer Associate (DP-203) preferred.
Expert level competencies in: Databrick Platform Administration (workspaces, cluster policies, SQL Warehouses), Unity Catalog & Data Governance (RBAC, Lineage, row/column-level security)
Advanced level competencies in: Azure Data Platform Integration (ALDS Gen2, Azure Networking, Entra ID/SCIM, Medallion Architecture (Bronze/Silver/Gold/ Delta/Lake), Monitoring, Cost Management & DR (System Tables, DBU tracking, RTO/RPO)
Identity & Access Management (Entra ID, SCIM, MFA, SoD)
Strong level competencies in: Infrastructure as Code (Terraform, Databricks, Asset Bundles)
CI/CD Pipelines & DevOps (Azure DevOps, branching strategy, SDLC)
Data Security & Compliance (data masking, audit logging, SOX controls)
Scripting & Automation (Python, PySpark, PowerShell)
Extensive experience with: Business Process Improvement
Decision Making and Critical Thinking
Global Perspective
Knowledge of Organization
Organizational Leadership
Resource Management

Responsibilities

Provision, configure, and manage Databricks workspaces in Azure, including cluster policies, SQL Warehouses, compute governance, and workspace-level security hardening.
Enforce secure cluster connectivity and no-policy cluster restrictions in alignment with EDW/ODS platform standards.
Administer and maintain Databricks Unity Catalog as the system of record for data access governance, including catalog/schema structure, RBAC/ACL policies, row-level and column-level security, data lineage registration, and metadata tagging.
Partner with the Manager, Database Engineering to enforce data governance standards and segregation of duties.
Manage Databricks user provisioning via SCIM/Automatic Identity Management integrated with Azure Active Directory/Entra ID.
Maintain access control matrices, conduct quarterly access reviews (QAR), and ensure MFA enforcement and local password disablement align with organizational policy.
Design and manage infrastructure provisioning using Terraform and Databricks Asset Bundles (DAB).
Support CI/CD pipeline promotion across Dev, QA, and Production environments via Azure DevOps.
Maintain repository structure and branching standards per SDLC guidelines.
Maintain and optimize the Bronze/Silver/Gold medallion zoning architecture within Databricks.
Ensure data retention policies, naming/tagging standards, and Unity Catalog object hierarchy align with enterprise data architecture standards.
Configure and maintain monitoring using Databricks System Tables, cluster and job-level alerting, and Azure Monitor integration.
Manage DBU consumption tracking, right-sizing, and cost guardrails.
Support backup and disaster recovery planning in alignment with defined RTO/RPO targets.
Work closely with the Cloud Engineering team to ensure Databricks infrastructure integrates seamlessly with Azure networking, storage (ADLS Gen2), and security components including ExpressRoute, VNets, and firewall rules.
Serve as the infrastructure handoff point to the Databricks Senior Data Engineer for pipeline and workload delivery.
Collaborate with cybersecurity and compliance teams to enforce data platform security controls including data masking/tokenization in non-production environments, audit logging, access reviews, and SOX/segregation of duties requirements.
Maintain accurate platform documentation and contribute to SDLC artifacts, change management templates, and runbooks for Databricks operations and incident response.
Stay current with evolving Databricks capabilities, Azure data platform developments, Unity Catalog enhancements, and data engineering best practices.
Recommend platform improvements that enhance reliability, cost-efficiency, and governance.
Completes other projects and duties as assigned.