Databricks Solutions Expert (Remote)

Govcio LLC • UNAVAILABLE, UNAVAILABLE

6d • Remote

About The Position

GovCIO is hiring a Databricks Solutions Expert to support the Department of Veterans Affairs (VA) Data Modernization initiative. This role involves architecting, designing, and implementing Azure Databricks as a multi-tenant, governed Lakehouse platform for thousands of analytics teams across the enterprise. The position blends architecture (platform, security, governance, networking) with engineering (data ingestion, transformation, streaming, ML lifecycle) to deliver a secure, scalable, and cost-efficient analytics foundation on Azure. This role is fully remote within the United States.

Requirements

• Bachelor’s degree in Information Technology or a related field (or commensurate experience).
• 12+ years of experience in data engineering/analytics.
• 5+ years building on cloud data platforms (Azure preferred).
• 3+ years hands-on Azure Databricks (platform + pipelines) and Delta Lake.
• Proven experience setting up Unity Catalog with granular governance (RLS/CLS).
• Deep knowledge of Azure networking (VNets, Private Link, NSGs), identity (Entra ID), Key Vault, ADLS Gen2, Event Hubs, Azure SQL/MI, and Data Factory.
• Strong Spark expertise (PySpark/SQL), Structured Streaming, performance tuning, partitioning and storage optimization.
• Practical Terraform experience for Databricks/Azure resources.
• Experience with CI/CD with Azure DevOps or GitHub Actions.
• Security-first mindset; track record implementing audit logging, policy-as-code, and compliance controls.
• Excellent communication skills; ability to standardize, teach, and influence at enterprise scale.
• Ability to obtain and maintain a suitability/Public Trust clearance.

Nice To Haves

• Experience operating platforms for >500 concurrent users and 1000s of analytics teams.
• Knowledge of Photon, DLT, Workflows, Lakehouse ML (MLflow, feature store), Delta Sharing.
• Exposure to FinOps and chargeback models for data platforms.
• Experience with Synapse/Fabric interoperability, and data virtualization patterns.
• Background in data modeling (medallion, dimensional, domain‑driven design) and data quality (expectations, SLAs).
• Databricks Certified Data Engineer Professional, Lakehouse Fundamentals, Machine Learning Associate/Professional certifications.
• Microsoft Azure Solutions Architect Expert (AZ‑305), Azure Data Engineer (DP‑203), Azure Security Engineer (AZ‑500) certifications.
• HashiCorp Terraform Associate certification.

Responsibilities

• Define platform patterns, build reference implementations, set standards (cluster policies, Unity Catalog governance, CI/CD), and drive enablement for product, analytics, and data science teams.
• Blueprint the Azure Databricks landing zone, including workspace topology, network architecture, and secure connectivity to data sources.
• Implement governance with Unity Catalog, covering metastores, catalog/schema/table design, data classification, security, lineage, and data sharing.
• Establish Lakehouse foundations with Delta Lake storage design, medallion data flow standards, partitioning, and performance best practices.
• Design for scalability to support thousands of teams through a multi-workspace strategy, tenancy model, cluster policies, and guardrails.
• Develop a reliability strategy including HA/DR, regional deployments, backup/restore, and repeatable environment provisioning.
• Implement security, compliance, and access control measures, including identity and access management, secrets management, data protection, and audit logging.
• Build DLT pipelines and jobs for batch and streaming data ingestion and transformation.
• Optimize pipeline performance, tune cluster sizing, and set standards for reliable jobs.
• Integrate MLflow for MLOps, including experiment tracking, model registry, and feature store.
• Implement end-to-end observability for jobs, clusters, and Unity Catalog audits, integrating with Azure Monitor.
• Develop reusable reference accelerators, run playbooks, and conduct office hours for team enablement.
• Provision workspaces, catalogs, cluster policies, and permissions via Infrastructure-as-Code (Terraform).
• Implement CI/CD pipelines for notebook/package deployment, testing, and environment promotion.
• Manage FinOps, including cost modeling, budgets, alerts, and usage analytics.
• Partner with Security, Networking, Compliance, and FinOps to codify enterprise standards.
• Establish a Lakehouse Platform Council to ratify patterns and review exceptions.
• Create adoption metrics, business case narratives, TCO models, and executive updates.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior