Databricks Platform Engineer

Haag, a Salas O'Brien Company•,

3h•$115,000 - $135,000

About The Position

The Databricks Platform Engineer / Architect owns the Databricks platform from end to end. You will administer the platform, keep it current, and advise teams across the organization on how to design and build within it effectively. This is a hands-on technical role with meaningful architectural authority. It is not a coordinator position. You will be responsible for both the daily operation of the platform and the long-term decisions that shape its scalability, security, governance, and performance.

Requirements

Databricks is your primary technical discipline, not simply one platform among many.
Several years of hands-on experience administering and architecting Databricks environments in production.
Equally comfortable writing Terraform code and participating in executive-level architecture, security, and governance discussions.
Ability to explain why a proposed approach may create challenges over time and offer practical alternatives.
Strong technical opinions and preferred approaches, but willingness to adjust them when evidence, business requirements, or organizational constraints support a different solution.
Communicate technical trade-offs clearly to people who are not platform engineers and do so without unnecessary jargon or condescension.
Deep, current, production-level Databricks administration experience on AWS, or significant Azure Databricks experience with a credible path to supporting AWS.
Experience designing and managing Unity Catalog in a multi-workspace environment.
Demonstrated experience developing catalog structures and access-control models that support a growing organization.
Experience integrating Databricks with Microsoft Entra ID, including single sign-on and SCIM provisioning.
Strong Terraform and CI/CD experience for infrastructure and platform management.
Working knowledge of cloud networking concepts, including virtual private clouds, private connectivity, encryption, egress controls, and related security-review processes.
Ability to build a strong partnership with IT Infrastructure and Security teams, clearly communicate platform needs, follow enterprise standards, and advocate constructively for architecture improvements.
Sound judgment in determining when the platform’s default approach is appropriate and when organizational requirements call for a different solution.
Strong written and verbal communication skills.

Nice To Haves

Databricks certifications, such as Databricks Certified Data Engineer Professional or Databricks platform administration credentials.
Experience with Databricks Asset Bundles.
Experience working in a federated, decentralized, or multi-entity organization where standardization requires partnership and influence rather than centralized mandates.
Experience with MLflow, machine learning operations, model serving, and related Databricks capabilities.
Experience supporting government, military, or controlled-data environments, including Controlled Unclassified Information and Cybersecurity Maturity Model Certification requirements.
Experience with enclave separation, controlled environments, and regulated-data operations.

Responsibilities

Administer the Databricks platform across multiple environments, including workspaces, Unity Catalog, cluster policies, budget policies, and the guardrails that enable teams to move quickly while maintaining consistency and control.
Manage identity and access, including Microsoft Entra ID single sign-on, account-level SCIM group provisioning, service principals, and scalable access-control models.
Partner with IT Infrastructure and Security to manage the platform’s networking and security posture, including customer-managed virtual private clouds, private connectivity, encryption, egress controls, audit logs, and system tables connected to enterprise monitoring tools.
Manage infrastructure and workloads through code using Terraform, Databricks Asset Bundles, and CI/CD practices, with no manual production configuration whenever automation is practical.
Establish and maintain cost-governance practices, including tagging standards, business-unit showback, budget controls, and recommendations for reducing unnecessary platform spending.
Serve as a trusted architecture advisor to data scientists, data engineers, analysts, and business stakeholders before solutions are designed and deployed.
Provide current, practical, and technically sound recommendations that balance organizational standards with business needs.
Evaluate emerging platform capabilities and determine how they should be adopted within the organization.
Monitor platform releases and understand how capabilities such as serverless computing, Unity Catalog, Lakeflow, and other emerging features affect our environment.
Translate new capabilities into practical recommendations rather than adopting technology based on hype.
Ensure that the organization uses modern, supportable patterns instead of relying on outdated approaches.