Senior Data Engineer

Five Forty•Arlington, TX

12d•Onsite

About The Position

540 is seeking a Senior Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will lead the design and evolution of Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale. Working with engineers, architects, cybersecurity teams, and mission stakeholders, you will translate complex requirements into secure, scalable solutions using Databricks, Apache Spark, and Delta Lake. You will define engineering standards, guide technical delivery, and mentor engineers while ensuring data quality, governance, and platform reliability.

Requirements

9+ years of relevant data engineering or software engineering experience
Experience leading enterprise-scale data platform and pipeline implementations using Databricks
Advanced proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake
Experience architecting large-scale ETL/ELT, batch, and streaming pipelines
Experience designing lakehouse architectures, data models, schemas, and data contracts
Experience managing and optimizing Databricks jobs, workflows, compute resources, and Spark workloads
Experience implementing data quality, monitoring, lineage, metadata management, and governance capabilities
Experience with Unity Catalog or similar data-governance and access-control solutions
Experience operating Databricks within AWS, Azure, or Google Cloud
Experience with CI/CD, infrastructure as code, automated testing, and source control
Strong understanding of data security, privacy, governance, and access-control principles
Experience leading technical reviews, mentoring engineers, and communicating architecture decisions

Nice To Haves

Relevant Databricks certification
Experience supporting DoW, federal, Advana, or other mission data environments
Experience building data platforms in classified, regulated, or mission-critical environments
Experience with Terraform, Databricks Asset Bundles, Airflow, or similar automation and orchestration tools
Experience architecting streaming solutions using Kafka, Kinesis, Pulsar, or Spark Structured Streaming
Experience supporting AI/ML pipelines, feature platforms, or MLflow
Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP

Responsibilities

Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities
Translate mission requirements into scalable data architectures and implementation strategies
Define data engineering standards, reusable patterns, and best practices across engineering teams
Architect automated ETL/ELT pipelines using Python, SQL, PySpark, Apache Spark, and Delta Lake
Design scalable data models, schemas, data contracts, and medallion architecture patterns
Lead the development of batch and streaming capabilities supporting operational, analytical, and AI/ML workloads
Establish data quality, lineage, metadata, observability, and governance practices using Unity Catalog or similar technologies
Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency
Establish CI/CD, infrastructure-as-code, testing, monitoring, and operational practices for Databricks environments
Lead design reviews and resolve complex issues spanning data pipelines, infrastructure, and production services
Partner with cybersecurity teams to incorporate security, access control, auditing, and governance requirements
Communicate architecture decisions and mentor engineers on Databricks and data engineering best practices