Data Engineer

Five Forty•Arlington, TX

12d•Onsite

About The Position

540 is seeking a Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will design, build, and maintain Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale. Working with software engineers, AI/ML engineers, cybersecurity teams, and mission stakeholders, you will build scalable and reliable solutions using Databricks, Python, Apache Spark, and Delta Lake. The ideal candidate enjoys solving complex engineering challenges and developing trusted data products that support national defense missions.

Requirements

4+ years of relevant data engineering or software engineering experience
Hands-on experience developing and operating production data pipelines using Databricks
Proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake
Experience building automated ETL/ELT pipelines for large-scale datasets
Experience designing and maintaining data models, schemas, tables, and lakehouse architectures
Experience managing Databricks notebooks, jobs, workflows, and compute resources
Experience implementing data quality, automated testing, monitoring, lineage, or metadata-management capabilities
Experience working with Databricks and cloud-based data services in AWS, Azure, or Google Cloud
Experience working with structured, semi-structured, and unstructured data
Understanding of lakehouse architecture, data governance, security, privacy, and access-control principles
Ability to troubleshoot data pipelines, Spark workloads, infrastructure, and applications

Nice To Haves

Databricks certification or equivalent demonstrated platform expertise
Experience supporting DoW, federal, Advana, or other enterprise data environments
Experience with CI/CD, infrastructure as code, automated testing, and source control
Experience using Unity Catalog for data governance, lineage, and access control
Experience developing streaming pipelines with Spark Structured Streaming, Kafka, Kinesis, or Pulsar
Experience with orchestration tools such as Airflow, Dagster, or Argo Workflows
Experience with Docker, Kubernetes, or other containerization and orchestration technologies
Experience building cloud-native data platforms in secure, regulated, classified, or mission-critical environments
Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP

Responsibilities

Design, develop, and maintain Databricks-based data pipelines, data products, and lakehouse capabilities
Build automated ETL/ELT pipelines that ingest, transform, and deliver mission-critical data
Develop production-grade data-processing solutions using Python, SQL, PySpark, Apache Spark, and Delta Lake
Design and maintain data models, schemas, tables, and medallion architecture patterns supporting analytical, operational, and AI/ML workloads
Build and operate batch and streaming data pipelines supporting mission requirements
Develop and manage Databricks notebooks, jobs, workflows, clusters, and compute resources
Implement data-quality checks, automated testing, monitoring, lineage, and metadata-management capabilities
Support data discovery, governance, and access controls using Unity Catalog or similar technologies
Optimize Spark workloads and Databricks resources for performance, scalability, reliability, and cost efficiency
Collaborate with engineers, analysts, and data scientists to deliver reusable data products and mission capabilities
Support Databricks deployments using CI/CD, infrastructure as code, and source control
Partner with cybersecurity teams to implement data-protection, access-control, auditing, and governance requirements
Troubleshoot issues affecting Databricks workloads, data pipelines, storage systems, and production data services
Document data models, pipeline designs, engineering processes, and operational procedures