Databricks Data Engineer, Lead

Booz Allen Hamilton • Bethesda, MD

1d • $77,600 - $176,000 • Hybrid

About The Position

As a lead Databricks Data Engineer, you will design, build, and maintain robust data pipelines within Databricks, contributing independently while providing technical leadership and mentorship. You will create and optimize end-to-end data flows, handle ETL processes, design data models, ensure data security and compliance, collaborate with cross-functional teams, and improve performance and scalability. The role uses Databricks and AWS expertise to enable efficient data processing and analytics.

You will mentor analysts, data engineers, developers, and data consumers in a fast-paced, agile environment. You will apply analytical exploration and data examination skills while overseeing the assessment, design, development, and maintenance of scalable client platforms. The position is hybrid, with work at a Booz Allen or customer facility in accordance with leadership expectations and role needs.

Requirements

• At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines, including comprehensive logging and robust error handling.
• At least 3 years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools.
• At least 3 years of experience with Databricks tools such as Delta Lake, Databricks Delta, and Unity Catalog.
• At least 3 years of experience writing Python and PySpark code for ETL scripts and custom transformations.
• Experience in data modeling, including selecting dimensional versus relational models within a Bronze, Silver, or Gold medallion architecture.
• Experience with AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
• Knowledge of ETL performance optimization techniques such as partitioning, bucketing, caching, and troubleshooting query performance.
• Ability to design and implement data models, from defining schemas to optimizing structures for queries and analytics.
• Public Trust determination is required; selected applicants are subject to a government investigation and may need to meet government-client eligibility requirements.
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology.

Nice To Haves

• Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch that support serverless data processing.
• Experience with AWS Glue Data Catalog or Collibra for metadata management, data lineage, and data asset cataloging.
• Knowledge of data governance, data lineage, and compliance requirements, including GDPR or CCPA, supporting data quality, privacy, and security.
• Ability to explain complex technical concepts to non-technical audiences, including architectural tradeoffs.

Responsibilities

• Design, build, and maintain robust data pipelines in Databricks.
• Create and optimize end-to-end data flows and ETL processes.
• Design data models and optimize data structures for efficient queries and analytics.
• Ensure data security and compliance.
• Collaborate with cross-functional teams.
• Improve data pipeline performance and scalability.
• Provide technical leadership and mentorship to the team.
• Mentor analysts, data engineers, developers, and data consumers.
• Oversee the assessment, design, building, and maintenance of scalable client platforms.
• Use analytical exploration and data examination to support data processing and analytics.
• Troubleshoot and resolve ETL and pipeline issues using logging and error-handling mechanisms.

Benefits

• Health, life, and disability benefits.
• Financial and retirement benefits.
• Paid leave.
• Professional development.
• Tuition assistance.
• Work-life programs.
• Dependent care.
• Recognition awards for exceptional performance and demonstration of company values.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior