Senior Data Engineer (Data Platform)

Gallatin•Washington, DC

2d•$80,000 - $210,000•Onsite

About The Position

Gallatin is hiring a Senior Data Engineer to build its Data Platform, which connects fragmented defense-logistics data sources, cleans and models the data, and serves it to Navigator and AI agents. The role designs and operates containerized services for external data connections, an open lakehouse, and transformations from raw source data to curated, semantically modeled entities. It also builds fullstack tools that help logistics subject-matter experts and analysts connect sources, map fields, and test pipeline changes. The platform must run across commercial and government clouds, single-tenant high-side enclaves, and air-gapped appliances at forward sites. The engineer will use open-source, Kubernetes-native components and hardened images, and design data movement for intermittent and disconnected networks. Success means mission partners can connect a new system of record and get validated, lineage-tracked, query-ready data into Navigator within days. The role is based in-office at one of Gallatin’s core engineering locations: El Segundo, California; Austin, Texas; Washington, DC; or San Francisco, California. It requires the ability to obtain and maintain relevant security clearances and to work in a classified environment when necessary.

Requirements

8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines and ownership of a data platform or lakehouse.
Production experience with an open table format (Apache Iceberg preferred; Delta or Hudi acceptable) and a distributed query engine (Trino, Presto, or Spark SQL), including partitioning, compaction, and schema evolution.
Experience with streaming and batch data pipeline technologies such as Kafka, NiFi, Debezium, or Spark.
Strong proficiency in Python and SQL, and experience integrating and validating data from heterogeneous relational and NoSQL databases, APIs, files, and streams.
Strong data modeling skills for analytical and operational workloads, including entity models for complex real-world domains.
Production orchestration experience with Airflow, Dagster, Argo Workflows, or similar.
Production experience writing services in Python and/or TypeScript, and frontend ability with Vue, React, or similar to deliver an internal tool end to end.
Ability to design and document RESTful APIs.
Strong Kubernetes and Docker experience, including packaging, deploying, and operating stateful data services on Kubernetes and debugging networking, storage, and namespace issues.
Comfort building with open-source tooling for self-hosted and air-gapped deployments.
Experience with AWS data services such as S3, Glue, or Aurora, or Azure equivalents, and containerized environments delivered through DevSecOps/CI-CD pipelines.
Ability to work independently on well-defined tasks with minimal oversight and communicate effectively with technical and non-technical stakeholders, including data providers and consumers.
Clear cross-disciplinary communication, including writing RFCs, explaining data models to subject-matter experts, or recording weekly demos.
Willingness to keep learning and stay current with industry trends and tools.
Ability to obtain and maintain relevant security clearances and work in a classified environment when necessary.

Nice To Haves

Active SECRET or TOP SECRET clearance.
Experience with data catalogs, lineage, and governance, such as Iceberg REST catalogs, Nessie, Polaris, OpenLineage, or DataHub.
Experience with data versioning and branching, such as Nessie or lakeFS, for sandboxed pipeline testing.
Apache NiFi or other low-code/no-code pipeline tools, including custom processors.
Experience in entity resolution, semantic data fusion, knowledge graphs, or ontology modeling.
Experience shipping data workloads into air-gapped or classified environments.
Palantir Foundry, Apollo, or ontology experience.
Data infrastructure for AI, including metadata and retrieval that make agents more accurate and explainable.
Operations research, optimization, or constraint-solver background.
Background in defense logistics, sustainment, or supply chain systems.
Contributions to open-source data projects.

Responsibilities

Design, build, and operate containerized services that create, configure, and manage external data connections, including databases, REST/SOAP APIs, file drops, message buses, and legacy systems of record.
Own the open lakehouse, including object storage, Apache Iceberg tables, a REST catalog with data versioning, and federated queries through Trino.
Build batch and streaming ingestion, including change data capture where appropriate, for disconnected, degraded, intermittent, and limited (DDIL) networks.
Run the platform on Kubernetes across commercial cloud, government cloud, and air-gapped edge appliances using hardened, FIPS-validated images.
Build transformation pipelines from raw through cleaned and curated layers into the semantic entity model used by Navigator and AI agents.
Implement entity resolution and data fusion so real-world entities resolve to a single record across sources.
Structure unstructured documents, slides, and spreadsheets into typed, relational records.
Build lineage, provenance, data-quality checks, and confidence labeling into pipelines.
Build fullstack applications and low-code workflows that let subject-matter experts connect sources, map fields, and compose pipelines.
Provide user sandboxes to branch data, test pipeline changes, compare results, and promote changes to production.
Build documented APIs consumed by Navigator features and AI agents.
Partner with Product, AI/ML, Frontend, and Infrastructure teams on RFCs, data models, and production code.
Work with mission partners and field teams to map data flows and translate operational constraints into platform requirements.
Contribute to code reviews, documentation, and CI/CD practices, and mentor engineers as the data team grows.