Data Engineer

FedWriters, Inc.•Arlington, MA

10d•$105,000 - $115,000•Onsite

About The Position

Ready to build data infrastructure that actually matters to national defense? As FWI's Data Engineer on the DA2I program, you'll architect end-to-end pipelines using Databricks, Spark, and Palantir Foundry that power decision-making for the National Guard's Minuteman Community Space. If you want your ETL and big-data skills to support a real DoW mission, not just another dashboard, this is the seat. FWI is building a team to support the National Guard Bureau's (NGB) Minuteman Community Space and as a Data Engineer, you will play a crucial role in designing, developing, and maintaining the Minuteman Community Space's data infrastructure and systems, using ETL, Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL to ensure efficient data processing and analysis.

Requirements

3-10 years of directly related experience and a B.A./B.S. or M.A./M.S. in a related field
3+ years of experience as a Data Engineer
Proficiency in ETL processes and firsthand experience with technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks
Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently
Understanding of data modeling/visualization, database design principles, and data governance practices
Must possess an active Secret clearance at minimum

Nice To Haves

Experience with the DoW mission and associated datasets
Previous experience with Advana or the War Data Platform
Experience with APIs
Experience with Machine Learning models and/or Generative AI
Experience building within Palantir Foundry applications, including Workshop, Ontology Manager, and Gaia

Responsibilities

Design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL
Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures
Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing
Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets
Leverage big data technologies to efficiently process and analyze large datasets
Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes
Troubleshoot data-related problems and provide innovative solutions to address complex data challenges
Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices