Databricks Data Engineer, Lead

Booz Allen Hamilton•Bethesda, MD

8d•$77,600 - $176,000

About The Position

As a lead Databricks Data Engineer, you will design, build, and maintain robust data pipelines in Databricks, contributing independently while providing technical leadership and mentorship. You will create and optimize end-to-end data flows, manage ETL processes, design data models, help ensure data security and compliance, collaborate across teams, and improve performance and scalability. Experience with Databricks and AWS will support your role as a technical team leader, bridging raw data and actionable insights for business data processing and analytics. You will mentor a multidisciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment. You will use analytical exploration and data examination to oversee the assessment, design, construction, and maintenance of scalable client platforms. Applicants selected will be subject to a government investigation and may need to meet U.S. government client eligibility requirements; a Public Trust determination is required. The projected annualized compensation range is $77,600 to $176,000. Salary depends on factors including location, education, skills, experience, and organizational requirements. The posting will close within 90 days of its Posting Date. The hiring process includes identity verification using biometrics and artificial intelligence. Candidates are expected to be on camera during interviews and assessments. Use of AI or other tools to assist with interview responses is prohibited unless explicitly permitted. Work model details depend on whether the role is listed as remote, hybrid, or onsite. Remote employees may occasionally need to work at a Booz Allen or customer facility; hybrid employees are expected to work at a Booz Allen or customer facility as directed by leadership and role needs; onsite work is primarily at a customer facility.

Requirements

8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines, including comprehensive logging and robust error handling.
3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools.
Experience with Databricks tools such as Delta Lake, Databricks Delta, and Unity Catalog.
3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations.
Experience in data modeling, including choosing dimensional versus relational data models within a medallion architecture (Bronze, Silver, or Gold).
Experience using AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
Knowledge of ETL performance optimization techniques such as partitioning, bucketing, caching, and troubleshooting query performance.
Ability to design and implement data models, from defining schemas to optimizing data structures for efficient queries and analytics.
Public Trust determination required.
Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology.

Nice To Haves

Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch for serverless data processing architectures.
Experience with AWS Glue Data Catalog or Collibra for metadata management, data lineage, and data asset cataloging.
Knowledge of data governance practices, data lineage, and compliance requirements, including GDPR or CCPA, supporting data quality, privacy, and security.
Ability to explain complex technical concepts to non-technical audiences, including architectural tradeoffs.

Responsibilities

Design, build, and maintain robust data pipelines within Databricks.
Create and optimize end-to-end data flows and handle ETL processes.
Design data models and data structures that support efficient queries and analytics.
Ensure data security and compliance.
Collaborate with cross-functional teams.
Continuously improve data pipeline performance and scalability.
Provide technical leadership and mentorship to team members.
Mentor analysts, data engineers, developers, and data consumers.
Oversee the assessment, design, building, and maintenance of scalable platforms for clients.
Troubleshoot and resolve issues using comprehensive logging and robust error handling mechanisms.