Data Engineer

West 4th Strategy•Arlington, VA

10d•$155,000 - $165,000•Hybrid

About The Position

We need an experienced Data Engineer at the National Guard Bureau (NGB). The NGB is the link between the Army and Air National Guard and the Department of War (DoW). You will support operations on the Minuteman project team, which manages the NGB's enterprise data environment within the DoW's central analytics platform, and plays a crucial role in designing, developing, and maintaining the Minuteman Community Space's data infrastructure and systems. In this role, you will design and implement end-to-end data pipelines using ETL processes and technologies including Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL, working closely with cross-functional teams to understand data requirements and design optimal data models and architectures that ensure efficient data processing and analysis. This is a full-time opportunity. We can offer a competitive salary, and a comprehensive benefits package.

Requirements

Three (3) to ten (10) years of directly related experience
Available to work onsite in Arlington, VA 1-2 days per week
3+ years of experience as a Data Engineer
Proficiency in ETL processes and firsthand experience with technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks
Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently
Understanding of data modeling/visualization, database design principles, and data governance practices
Active Secret security clearance
U.S. citizenship is required since this role supports the U.S. Federal government

Nice To Haves

Experience with the Department of War (DoW) mission and associated datasets
Previous experience with Advana or the War Data Platform
Experience with APIs
Experience with Machine Learning models and/or Generative AI
Experience building within Palantir Foundry applications, including Workshop, Ontology Manager, and Gaia

Responsibilities

Design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL
Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures
Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing
Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets
Leverage big data technologies to efficiently process and analyze large datasets
Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes
Troubleshoot data-related problems and provide innovative solutions to address complex data challenges
Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices
Research, design, and develop software and specialized utility programs in support of mission requirements
Analyze user needs and develop software solutions, applying principles and techniques of computer science, engineering, and mathematical analysis
Update and enhance existing software capabilities
Develop specifications and performance requirements, coordinating with engineering counterparts to integrate hardware and software systems as required
Maintain databases within the application area, working individually or coordinating database development as part of a team