Data Engineer (Streaming & Sensor Data)

Addx Corporation • Colorado Springs, CO

2d

About The Position

We are seeking a Data Engineer to build the data foundation for machine learning applications. The role integrates time-series measurements, including radar and OPIR data, from real-time streams, application databases, and systems with varying extraction capabilities. The engineer will collaborate with senior technical staff and the ML team to characterize available data, establish reliable pipelines, and deliver cleaned, standardized datasets. They will help select technologies and develop an architecture based on measured workloads, application needs, and operational requirements. The development environment is primarily Python, with AWS as the preferred cloud platform.

The position involves investigating data sources and their interfaces, message formats, historical coverage, volumes, arrival rates, and quality limitations; building maintainable ingestion workflows for streaming and historical data; evaluating processing, storage, and access options; delivering trusted datasets with consistent schemas and traceability; and operating tested, documented pipelines with repeatable deployment, monitoring, access controls, and recovery and reprocessing procedures.

Requirements

• Experience designing, implementing, and operating production streaming and batch data pipelines and supporting storage systems.
• Strong software engineering skills, including maintainable code, automated testing, version control, and deployment.
• Practical understanding of streaming behavior, including filtering, consumer progress, buffering, failure recovery, and replay.
• Experience organizing time-series, telemetry, sensor, or similarly complex data, including understanding schema evolution, partitioning, and efficient access patterns.
• Ability to investigate unfamiliar systems, evaluate technical tradeoffs, and communicate recommendations while balancing reliability, performance, maintainability, and cost.

Nice To Haves

• Python-based data processing and AWS or comparable cloud infrastructure.
• Radar, OPIR, or other sensor measurement data.
• Workflow orchestration, containers, infrastructure as code, and automated delivery.
• Dataset versioning, feature generation, labeling, or training-data preparation.

Responsibilities

• Investigate data source interfaces, message formats, historical coverage, data volumes, arrival rates, and quality limitations, and document findings to guide implementation and capacity planning.
• Develop maintainable ingestion workflows for streaming and historical data across databases, APIs, files, and application-specific interfaces.
• Implement configurable filtering to capture messages and measurements needed downstream.
• Evaluate processing, storage, and access options against representative workloads, reliability needs, interoperability, operational complexity, and cost; document decisions and support architecture evolution.
• Establish consistent dataset schemas, timestamps, units, identifiers, and source metadata while preserving measurement meaning.
• Implement data validation and traceability, and define dependable access methods with the ML team.
• Support future feature, labeling, and training-data workflows.
• Build tested and documented pipelines with repeatable deployment, monitoring, and appropriate access controls.
• Handle duplicate, missing, delayed, and out-of-order data, and implement recovery and reprocessing procedures.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed