Sr. Databricks Data Engineer

Halvik • McLean, IL

6d

About The Position

The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation (DOT) data modernization effort by designing, building, migrating, and operating secure, scalable data pipelines in a Databricks Lakehouse. This is a highly hands-on role focused on legacy and external-source migration, metadata-driven ingestion, Medallion Architecture, data quality and reconciliation, production operations, and transition to the AWS/OneDOT environment.

Requirements

• Bachelor's degree in computer science, Data Engineering, Information Technology, Engineering, or a related field; equivalent relevant experience may be considered.
• Minimum of 7 years of professional data engineering experience, including significant hands-on experience with Databricks/Spark and production data pipelines.
• Advanced Python, PySpark, and SQL skills; strong experience with Databricks, Delta Lake, AWS/S3, batch/file ingestion (CSV/JSON), REST APIs, Git/CI/CD, and data pipeline orchestration.
• Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data, including access controls and secrets management.

Nice To Haves

• Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands-on experience will be considered.
• Demonstrated success modernizing legacy data platforms (Sybase/SAP IQ or similar) into Databricks, including source-to-target reconciliation, cutover, and production stabilization.
• Experience with Unity Catalog, metadata/configuration-driven ingestion, Auto Loader/Lakeflow, Databricks Jobs/Workflows, lineage/cataloging tools, and scalable multi-source ingestion frameworks.
• Federal/DOT delivery experience; familiarity with Agile/Kanban, production O&M, cloud cost optimization, and data governance.
• Streaming and AI/ML data pipeline exposure is a plus.

Responsibilities

• Lead the design and implementation of reusable ingestion and transformation patterns for legacy and external data sources into Databricks, including support for migration to the AWS/OneDOT environment.
• Build and operate Bronze, Silver, and Gold pipelines; manage batch and incremental loads, orchestration, dependencies, retries/recovery, monitoring, alerting, and production support.
• Partner with the Technical Project Manager, Data Architects, business/data SMEs, BI developers, Cloud/DevSecOps, security, and governance teams to translate requirements into implementable data solutions.
• Develop and optimize Python, PySpark, and SQL workloads using Databricks, Delta Lake, Unity Catalog, Auto Loader and/or Lakeflow capabilities, and Databricks Jobs/Workflows with a focus on reliability, performance, and cost.
• Implement source-to-target reconciliation, data validation, schema evolution, lineage, access controls, documentation, and reusable engineering standards for production data pipelines.

Benefits

• Company-supported medical, dental, vision, life, STD, and LTD insurance
• 11 federal holidays and PTO
• Eligible employees may receive performance-based incentives in recognition of individual and/or team achievements.
• 401(k) with company matching
• Flexible Spending Accounts for commuter, medical, and dependent care expenses
• Tuition Assistance
• Charitable Contribution matching

What This Job Offers

• Job Type: Full-time
• Career Level: Senior