Databricks Data Engineer and Scientist

Booz Allen Hamilton • Bethesda, MD

2d • $77,600 - $176,000

About The Position

As a Databricks Data Engineer and Scientist, you will design, build, and maintain robust data pipelines for a critical enterprise Lakehouse, applying data science, AI, and analytics skills. You will develop and optimize end-to-end data flows, manage ETL processes, design data models, help ensure data security and compliance, collaborate with cross-functional teams, and improve performance and scalability. Using Databricks, AWS, SQL, Python, and AI-assisted coding skills, you will analyze pipelines, support data modeling, and turn raw data into actionable business insights.

Requirements

• 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines, with comprehensive logging and error handling.
• 3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.
• 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations.
• Experience using AI-assisted coding tools such as Claude Code or Cursor for data engineering and data modeling tasks.
• Experience with data modeling, including selecting dimensional or relational models, within a Bronze, Silver, and Gold medallion architecture.
• Experience using AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
• Knowledge of ETL performance optimization techniques such as partitioning, bucketing, and caching, and experience troubleshooting query performance.
• Ability to analyze and decompose complex legacy SQL, rebuild data pipelines or models, and compare results to validate accuracy and performance.
• Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Math, Physics, or Information Systems and Technology.

Nice To Haves

• Experience with AWS cloud-native data services such as AWS Lambda or Amazon CloudWatch that support serverless data processing.
• Experience with metadata and data cataloging tools such as AWS Glue Data Catalog or Collibra to manage data assets and lineage.
• Knowledge of data governance, data lineage, and compliance requirements such as GDPR and CCPA that support data quality, privacy, and security.
• Ability to explain complex technical concepts to non-technical audiences, including architectural tradeoffs.

Responsibilities

• Design, develop, deploy, and manage ETL processes and data pipelines, including logging and error handling.
• Build, debug, and optimize pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.
• Write Python and PySpark ETL scripts and custom transformations.
• Design data models, choosing dimensional or relational models within a Bronze, Silver, and Gold medallion architecture.
• Use AWS services including AWS Glue, Amazon S3, and Amazon RDS.
• Optimize ETL performance using techniques such as partitioning, bucketing, and caching, and troubleshoot query performance.
• Analyze complex legacy SQL, rebuild data pipelines or models, and compare results to validate accuracy and performance.
• Collaborate with cross-functional teams to support data modeling and turn raw data into actionable insights.
• Help ensure data security and compliance and improve pipeline performance and scalability.

Benefits

• Health, life, disability, financial, and retirement benefits.
• Paid leave.
• Professional development.
• Tuition assistance.
• Work-life programs.
• Dependent care.
• Recognition awards for exceptional performance and demonstration of company values.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior