Software Engineer - Data Platform (Staff)

Deterrence Defense, Inc.•Fremont, CA

9h•$195,000 - $300,000•Onsite

About The Position

Deterrence builds autonomous production lines for energetics and munitions. Those lines generate process data that cannot be bought or synthesized: millions of signals per facility, tied to individual parts, kept as quality evidence for decades and used to train our models. The Staff Data Platform Engineer owns how that data moves from the factory edge into AWS GovCloud, how it is kept permanently, and how people and models read it: a complete, never-deleted lakehouse record on one path, and a sub-second live feed for line and customer dashboards on the other. The ideal candidate has run production data infrastructure where losing or mislabeling data was not an option, treats correctness and tenant isolation as properties to be tested rather than assumed, and wants their work to be the system of record for real factories.

Requirements

Bachelor's degree in Computer Science, Computer Engineering or a related field, or equivalent experience.
8+ years building production data infrastructure or distributed systems, including on-call ownership and recovery of a system where data loss was unacceptable.
Deep hands-on experience with a lakehouse table format (Iceberg, Delta Lake or Hudi) on object storage: partitioning, compaction, snapshot management, schema evolution and row-level deletes.
Strong production Python and advanced SQL; comfortable in a second language (Go, Rust, TypeScript, Java or Scala).
Time-series data at scale: high cardinality, late and out-of-order data, bucketing and downsampling, source time versus arrival time.
Distributed-systems correctness in practice: idempotence, retries, replay, backpressure, at-least-once delivery with de-duplication, and durable buffering over unreliable links.
Hands-on AWS data services (S3, Athena, Glue, IAM) and operation of OpenSearch or Elasticsearch: index and shard design, aliases, refresh tuning.
PostgreSQL depth: partitioning, row-level security, query performance.
Infrastructure as code (Terraform or CDK) and CI/CD for data pipelines.

Nice To Haves

AWS GovCloud or other regulated environments: ITAR, CUI, NIST 800-171, CMMC or FedRAMP.
Multi-tenant data platforms with server-side isolation.
ML data infrastructure: training datasets, feature pipelines, SageMaker or similar.
Streaming at scale (Kinesis, Kafka or MQTT) and edge or intermittently connected systems (AWS IoT Core, Greengrass).
Manufacturing, quality or traceability data; hardware-adjacent domains.
Cost engineering of data platforms: knowing what a query, a message and a stored byte cost, and designing to it.

Responsibilities

Own the data and telemetry architecture end to end: the bulk path from edge historians to an S3 and Apache Iceberg lake, and the serving path from MQTT to a low-latency search store and live dashboards.
Build edge-to-cloud export that survives outages: durable edge buffers, export–confirm–delete semantics, idempotent landing, backlog replay, and disk guards, so that a cloud outage never stops a production cell.
Run the lakehouse: raw-to-Iceberg conversion, partitioning by tenant and date, compaction and snapshot maintenance, schema evolution, storage tiering, and audited purges in a store whose default is never delete.
Own signal identity and data contracts: a registry that gives every signal a stable ID, unit and conversion; source-time semantics; clock-drift detection; and canaries that prove the live and historical views agree.
Make multi-tenant isolation provable across index aliases, IAM prefix policies and PostgreSQL row-level security, with CI tests that fail if one customer can read another's data.
Deliver the datasets the business runs on: SQL that joins process telemetry to part genealogy for quality investigations, training and feature datasets for ML, and cost-bounded summaries for dashboards.
Bring inspection results, images and video into the same record, correlated to the machine jobs and parts that produced them.
Measure before scaling: signal change rates, historian and edge-buffer capacity, and per-tenant query cost, turned into capacity and cost models.
Write and review the architecture decision records for each data component, mentor engineers, and lead technical work across the edge, controls, traceability and front-end teams.