Senior Data Engineer

General Dynamics Information Technology • Fort Bragg, NC

10d • Hybrid

About The Position

We are seeking a Data Engineer SME with a strong ETL emphasis to design, build, and maintain data collection, processing, and integration pipelines that centralize and structure data from various web-based applications. You will work closely with software developers, data analysts, and product teams to ensure seamless data flow, high-quality data storage, and efficient retrieval methods. This role is ideal for someone passionate about data-driven decision-making and optimizing data aggregation processes while exploring and identifying right-sized technologies and computing resources. This position focuses on wrangling disparate data sources into unified, consumable data APIs for internal and external stakeholders. This is a hybrid position (3 days on site) in Fort Bragg, NC.

Requirements

• Extensive experience with Apache Airflow for workflow orchestration.
• Strong programming skills in Python.
• Experience using SQL compliant database solutions such as Postgres.
• Familiarity with Trino for enhanced SQL query processing in large datasets.
• Experience with ElasticSearch/OpenSearch for data indexing and search functionalities.
• Understanding of vector databases, embedding models, and vector search for AI applications.
• Expertise in event-driven architecture and microservices development.
• Hands-on experience with cloud services, particularly AWS S3 and MinIO, including data storage and compute resources.
• Strong knowledge of Reggo and its application within data environments.
• Understanding of big data technologies and file formats like Parquet.
• Expertise in data pipeline orchestration and workflow automation.
• Working knowledge of Linux environments and database optimization techniques.
• Strong understanding of version control with Git.
• Familiarity with OpenMetaData for data management.
• Experience implementing FastAPI for building scalable APIs.
• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
• An active TS/SCI security clearance is REQUIRED, and candidates must have or be willing to obtain a CI Poly.
• Only US Citizens are eligible for this role.

Nice To Haves

• Advanced degrees are a plus.

Responsibilities

• Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow.
• Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
• Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
• Implement and optimize Python-based data processing solutions.
• Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
• Ensure adherence to security and compliance requirements when working with classified data.
• Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
• Design, implement, and maintain robust data engineering solutions serving data to the backend of production user facing applications.
• Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
• Optimize data storage, retrieval, and processing efficiency, focusing on big data and the use of Parquet files.
• Utilize Trino for distributed SQL query processing.
• Leverage OpenMetaData for data discovery and governance.
• Implement APIs using FastAPI for streamlined data interaction.
• Ensure proper implementation of Data Governance requirements and adherence to common PKI security practices to appropriately protect data.
• Leverage IC repositories of user attributes to provide authorized access to data and ensure proper protections are applied.

Benefits

• 401K with company match
• competitive pay
• paid time off
• full flex work weeks where possible
• variety of medical plan options
• dental plan options
• vision plan
• short and long-term disability benefits
• life, accidental death and dismemberment, personal accident, critical illness and business travel and accident insurance

What This Job Offers

• Job Type: Full-time
• Career Level: Senior