Databricks Engineer

White & Case LLP•Tampa, FL

About The Position

The Databricks Engineer will join an agile delivery team building and supporting the Firm’s enterprise data platform on Azure Databricks. The role designs, develops, tests, and operates scalable data pipelines and curated data products that integrate information from on-premises, cloud, and third-party systems. The engineer works with business analysts, architects, application teams, and other stakeholders to turn business requirements into reliable technical solutions, delivering work through Azure DevOps and Microsoft GitHub. Clear communication is important for explaining data designs, pipeline behavior, data quality, and delivery decisions to technical and business audiences. The Business Intelligence group is the Firm’s center of excellence for data analytics, data management, and data governance. It develops and maintains datasets, dashboards, and reports, and collaborates across functions to provide reliable, accessible insights that support strategic decisions and operational efficiency. The Databricks Engineer contributes to trusted data products for analytics, reporting, and AI, as well as to enterprise data governance and engineering standards.

Requirements

At least five years of experience delivering production data engineering, data warehouse, or business intelligence solutions.
Bachelor’s degree in technology, engineering, computer science, or a related field, or an equivalent combination of education and experience.
Hands-on experience with Azure Databricks, Apache Spark, Delta Lake, and lakehouse architecture.
Strong SQL skills and practical experience with Python or PySpark for data transformation and pipeline development.
Experience designing and supporting batch data ingestion, ELT processes, orchestration, error handling, monitoring, and data quality controls.
Strong knowledge of data architecture, dimensional modeling, star schemas, query design, and performance optimization.
Experience integrating relational databases, APIs, files, cloud applications, and other enterprise data sources.
Working knowledge of Unity Catalog, data governance, access controls, lineage, and secure data handling.
Experience with Git-based development, automated testing, and CI/CD practices using GitHub, GitHub Actions, Azure DevOps, or comparable tooling.
Familiarity with relevant Azure services such as Azure Data Lake Storage, Azure Data Factory, Key Vault, and networking or identity services.
Knowledge of software development life cycle concepts and agile delivery methods.
Strong analytical, troubleshooting, and communication skills, including the ability to explain technical concepts clearly to business and technical stakeholders.
Ability to work independently and collaboratively, manage competing priorities, and deliver reliable solutions within established timelines.

Nice To Haves

Exposure to semantic models, business ontologies, agentic AI patterns, or standards such as Model Context Protocol (MCP).

Responsibilities

Design, build, test, deploy, and support scalable data pipelines and data products on Azure Databricks.
Develop ingestion and transformation processes for on-premises, cloud, SaaS, and third-party data using batch and, where appropriate, streaming patterns.
Use SQL and Python or PySpark to transform, validate, and optimize large and complex datasets.
Design and maintain reusable data structures across raw, refined, and curated layers, applying dimensional modeling and star schema principles where appropriate.
Build reliable Delta Lake solutions and use Databricks capabilities for orchestration, performance, observability, and data quality.
Manage and govern data assets through Unity Catalog, including access controls, lineage, documentation, and secure handling of sensitive information.
Apply software engineering practices including source control, code review, automated testing, reusable components, and CI/CD; help develop coding standards, patterns, and templates.
Monitor production workloads, troubleshoot failures and performance issues, and improve platform reliability, efficiency, and cost effectiveness.
Partner with business analysts, architects, application teams, and subject-matter experts to define requirements, source data, technical designs, and delivery plans.
Translate business concepts into trusted, well-documented data products that support analytics, reporting, and AI use cases.
Share knowledge and help plan and lead technical sessions that strengthen the team’s Databricks and data engineering capabilities.
Contribute to data governance, architecture standards, and engineering practices that improve enterprise data quality, consistency, security, and availability.
Stay current on Azure Databricks and related Azure capabilities and recommend practical improvements to the platform and delivery approach.