Senior Data Engineer

General Dynamics Information Technology•USA NC Fort Bragg - 2744 Malvesti St (NCC075), NC

12d•$168,300 - $227,700•Hybrid

About The Position

We are seeking a Data Engineer SME with a strong ETL emphasis to design, build, and maintain data collection, processing, and integration pipelines that centralize and structure data from various web-based applications. You will work closely with software developers, data analysts, and product teams to ensure seamless data flow, high-quality data storage, and efficient retrieval methods. This role is ideal for someone passionate about data-driven decision-making and optimizing data aggregation processes while exploring and identifying right-sized technologies and computing resources. This position focuses on wrangling disparate data sources into unified, consumable data APIs for internal and external stakeholders. This is a hybrid position (3 days on site) in Fort Bragg, NC.

Requirements

Extensive experience with Apache Airflow for workflow orchestration.
Strong programming skills in Python.
Experience using SQL compliant database solutions such as Postgres.
Familiarity with Trino for enhanced SQL query processing in large datasets.
Experience with ElasticSearch/OpenSearch for data indexing and search functionalities.
Understanding of vector databases, embedding models, and vector search for AI applications.
Expertise in event-driven architecture and microservices development.
Hands-on experience with cloud services, particularly AWS S3 and MinIO, including data storage and compute resources.
Strong knowledge of Reggo and its application within data environments.
Understanding of big data technologies and file formats like Parquet.
Expertise in data pipeline orchestration and workflow automation.
Working knowledge of Linux environments and database optimization techniques.
Strong understanding of version control with Git.
Familiarity with OpenMetaData for data management.
Experience implementing FastAPI for building scalable APIs.
Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
An active TS/SCI security clearance is REQUIRED, and candidates must have or be willing to obtain a CI Poly.
Due to US Government Contract Requirements, only US Citizens are eligible for this role.

Nice To Haves

Advanced degrees are a plus.

Responsibilities

Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow.
Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
Implement and optimize Python-based data processing solutions.
Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
Ensure adherence to security and compliance requirements when working with classified data.
Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
Design, implement, and maintain robust data engineering solutions serving data to the backend of production user facing applications.
Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
Optimize data storage, retrieval, and processing efficiency, focusing on big data and the use of Parquet files.
Utilize Trino for distributed SQL query processing.
Leverage OpenMetaData for data discovery and governance.
Implement APIs using FastAPI for streamlined data interaction.
Ensure proper implementation of Data Governance requirements and adherence to common PKI security practices to appropriately protect data.
Leverage IC repositories of user attributes to provide authorized access to data and ensure proper protections are applied.

Benefits

Comprehensive benefits and wellness packages
401K with company match
competitive pay
paid time off
variety of medical plan options, some with Health Savings Accounts
dental plan options
a vision plan
a 401(k) plan offering the ability to contribute both pre and post-tax dollars up to the IRS annual limits and receive a company match
full flex work weeks where possible
a variety of paid time off plans, including vacation, sick and personal time, holidays, paid parental, military, bereavement and jury duty leave
short and long-term disability benefits
life, accidental death and dismemberment, personal accident, critical illness and business travel and accident insurance