Senior Databricks Engineer

CNX•USA, IA, Work-at-Home, IA

1d•$106,087 - $151,554•Remote

About The Position

The Senior Data Engineer — Databricks owns the end-to-end data lifecycle, from analysis and solution design through implementation, orchestration, governance, and optimization. The role builds modern data solutions on the Databricks Data Intelligence Platform across Microsoft Azure, AWS, or GCP, using Databricks, Python, PySpark, and SQL to deliver scalable ingestion, transformation, modeling, and pipeline orchestration. As a senior engineer, this position leads solution design, establishes engineering standards, and guides other data engineers in building reliable, governed, production-ready data pipelines that support analytics, reporting, and AI/ML use cases. The role is remote/work at home in the United States. The position is an existing, immediate vacancy, with a desired start as soon as possible.

Requirements

10 or more years of professional experience in data engineering, data and analytics, or data warehousing.
5 or more years of hands-on Databricks experience in a data engineering capacity, including production workloads.
5 or more years of Python experience, including manipulating large datasets with PySpark DataFrames, Spark SQL, and/or the pandas API on Spark.
3 or more years of data engineering experience on Microsoft Azure, including services such as ADLS Gen2, Azure Data Factory, Key Vault, and Microsoft Entra ID.
Strong experience designing and developing ETL/ELT pipelines for batch and streaming data workloads.
Experience with Databricks lakehouse architecture, Delta tables, Unity Catalog, Lakeflow Jobs, and Lakeflow Spark Declarative Pipelines.
Experience implementing scalable data models using medallion architecture, dimensional modeling, and slowly changing dimensions.
Experience with CI/CD and data pipeline testing practices using Azure DevOps or GitHub Actions.
Ability to lead technical design discussions, conduct code reviews, define engineering standards, and mentor other engineers.
Strong collaboration skills and ability to work across business, IT, and technical stakeholder groups.
Must be legally authorized to work in the United States and reside in the United States or have a valid U.S. address for residence.

Nice To Haves

Databricks Data Engineer Professional certification.

Responsibilities

Design, develop, and maintain batch and streaming data pipelines that extract, transform, and load data from multiple sources into the Databricks lakehouse on Azure.
Automate and streamline data ingestion using Auto Loader, Lakeflow Connect, Azure Data Factory, and Event Hubs.
Build and manage Lakeflow Spark Declarative Pipelines, including data quality expectations and change data capture.
Use PySpark and Spark SQL for complex data manipulation and transformation supporting analytics, reporting, and AI/ML use cases.
Design and implement workflow orchestration and scheduling with Lakeflow Jobs.
Create scalable lakehouse data models using medallion architecture and dimensional modeling, including SCD Type 1 and Type 2.
Implement governance and security in Unity Catalog, including access controls, lineage, row filters, and column masks for sensitive data.
Optimize performance and cost across Delta tables, Spark workloads, serverless and classic compute, and SQL warehouses.
Establish CI/CD and testing practices for data pipelines using Git folders, Declarative Automation Bundles, Azure DevOps, or GitHub Actions.
Monitor pipeline health and data quality using system tables and data quality monitoring; lead root-cause analysis and resolution of production issues.
Lead design and code reviews, define engineering standards, and mentor data engineers.
Collaborate with business stakeholders, IT, and cross-functional teams to ensure data solutions meet business, performance, and project requirements.