Sensitive-Data Engineer

Equilibrium Technologies

55d

About The Position

Are you passionate about harnessing data to solve some of the nation’s most critical challenges? Do you thrive on innovation, collaboration, and building resilient solutions in complex environments? Join a high-impact team at the forefront of national security, where your work directly supports mission success. We're seeking a Data Engineer with a rare mix of curiosity, craftsmanship, and commitment to excellence. In this role, you’ll design and optimize secure, scalable data pipelines while working alongside elite engineers, mission partners, and data experts to unlock actionable insights from diverse datasets.

Requirements

Active Top Secret/SCI Clearance with Polygraph (REQUIRED)
Experience building and maintaining data pipelines using Apache Spark, Airflow, NiFi, or dbt
Proficiency in Python, SQL, and one or more of: Java, Scala
Strong understanding of cloud services (especially AWS and GovCloud), including S3, EC2, Lambda, EMR, Glue, Redshift, or Snowflake
Hands-on experience with streaming frameworks such as Apache Kafka, Kafka Connect, or Flink
Familiarity with data lakehouse formats (e.g., Apache Hudi, Delta Lake, or Iceberg)
Experience with NoSQL and RDBMS technologies such as MongoDB, DynamoDB, PostgreSQL, or MySQL
Ability to implement and maintain data validation frameworks (e.g., Great Expectations, Deequ)
Comfortable working in Linux/Unix environments, using bash scripting, Git, and CI/CD tools
Knowledge of containerization and orchestration tools like Docker and Kubernetes
Collaborative mindset with experience working in Agile/Scrum environments using Jira, Confluence, and Git-based workflows

Responsibilities

Engineer robust, secure, and scalable data pipelines using Apache Spark, Apache Hudi, AWS EMR, and Kubernetes
Maintain data provenance and access controls to ensure full lineage and auditability of mission-critical datasets
Clean, transform, and condition data using tools such as dbt, Apache NiFi, or Pandas
Build and orchestrate repeatable ETL workflows using Apache Airflow, Dagster, or Prefect
Develop API connectors for ingesting structured and unstructured data sources
Collaborate with data stewards, architects, and mission teams to align on data standards, quality, and integrity
Provide advanced database administration for Oracle, PostgreSQL, MongoDB, Elasticsearch, and others
Ingest and analyze streaming data using tools like Apache Kafka, AWS Kinesis, or Apache Flink
Perform real-time and batch processing on large datasets in secure cloud environments (e.g., AWS GovCloud, C2S)
Implement and monitor data quality and validation checks using tools such as Great Expectations or Deequ
Work across agile teams using DevSecOps practices to build resilient full-stack solutions with Python, Java, or Scala