Data Engineer / Enterprise Telemetry & Data Pipeline SME

Govcio LLC•San Antonio, TX

7d•Onsite

About The Position

GovCIO is seeking a Senior Data Engineer / Enterprise Telemetry & Data Pipeline SME to design, optimize, automate, and sustain large-scale data collection and streaming pipelines supporting Air Force Intelligence Community missions. The environment includes a global Elastic deployment with distributed endpoint collectors, syslog and telemetry from Cisco and other network devices, Cribl-based routing and normalization of native and non-native sources, Splunk engineering and integration, and AWS-based infrastructure and services. The primary focus is the movement and optimization of machine-generated data from source to authorized analytics and SIEM destinations. This is not primarily a data science or dashboard-development role. The successful candidate must be able to evaluate the complete data path, identify collection and transport inefficiencies, engineer scalable solutions, automate repeatable operations, and independently resolve complex ingestion, routing, performance, reliability, and integration problems. This position will be located on Lackland AFB and will be an onsite position.

Requirements

High School with 10+ years (or commensurate experience)
Candidates must demonstrate substantial hands-on experience engineering production data pipelines. Data analysis, dashboard development, SIEM use, or product administration without ownership of collection and transport architecture is not sufficient for this position.
Clearance: TS/SCI
10+ years of progressively responsible experience in data engineering, platform engineering, DevOps, systems engineering, software engineering, observability engineering, or closely related disciplines.
5+ years of hands-on experience designing, implementing, operating, or optimizing large-scale production data ingestion, logging, telemetry, event-streaming, or observability pipelines.
Demonstrated experience serving as a senior engineer, technical lead, architecture resource, primary escalation point, or technical owner for production data platforms or pipelines.
Strong understanding of distributed data-pipeline architecture, including collection, transport, buffering, queueing, parsing, transformation, normalization, enrichment, routing, storage, and downstream consumption.
Demonstrated ability to diagnose and optimize throughput, latency, reliability, data quality, bandwidth utilization, processing efficiency, and pipeline resiliency.
Significant hands-on production experience with Elastic ingestion and collection technologies such as Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, or comparable mechanisms.
Hands-on production experience with Cribl Stream, Cribl Edge, or equivalent enterprise routing/normalization technology, including building and troubleshooting sources, routes, pipelines, transformations, and destinations.
Hands-on experience engineering or supporting Splunk ingestion and collection workflows using forwarders, syslog, HTTP Event Collector, APIs, or equivalent collection mechanisms.
Experience ingesting and processing syslog, endpoint telemetry, network-device data, application logs, structured/unstructured events, API data, and other machine-generated sources.
Strong knowledge of parsing, field extraction, schema design, normalization, enrichment, filtering, transformation, and routing.
Demonstrated production experience with Amazon Web Services (AWS) and ability to design and troubleshoot AWS-based data-processing and telemetry architectures.
Working knowledge of AWS services and patterns relevant to streaming, processing, storage, and monitoring, such as Kinesis, MSK/Kafka, S3, Lambda, EKS/ECS, CloudWatch, IAM, and VPC services.
Strong Python programming and scripting experience used for automation, APIs, data manipulation, validation, testing, or engineering tools.
Strong Linux systems engineering and troubleshooting skills and working knowledge of enterprise networking, including TCP/IP, DNS, routing, segmentation, TLS, proxies/firewalls, bandwidth, latency, and connectivity troubleshooting.
Demonstrated use of Git for source control, peer review, branching, configuration management, and collaborative engineering workflows.
Demonstrated experience with CI/CD or DevSecOps pipelines using GitLab CI/CD, Jenkins, GitHub Actions, AWS CodePipeline, Azure DevOps, or comparable technologies.
Demonstrated Infrastructure as Code experience using Terraform, AWS CloudFormation, or equivalent technologies; experience with Ansible or comparable configuration automation.
Working knowledge of cybersecurity requirements applicable to enterprise telemetry, including encryption, authentication, authorization, certificates, secrets, access control, data integrity, secure configuration, and change control.
Demonstrated ability to independently investigate complex data-flow or integration problems, determine root cause, design remediation, implement and validate the solution, and document the outcome.
Strong technical documentation and communication skills with the ability to coordinate effectively with Government stakeholders, cybersecurity, network, cloud, systems, DevSecOps, and analytics teams.
Must meet IAT II 8570 certification requirement in lieu of defined 8140.2 defined work role.

Nice To Haves

Direct production experience with Elastic, Splunk, and Cribl operating together within the same enterprise telemetry architecture.
Advanced Elastic ingestion experience, including large-scale Agent/Fleet, Logstash, data streams, index templates, lifecycle management, cluster ingestion behavior, and performance troubleshooting.
Advanced Cribl engineering, including Stream/Edge, worker groups, routes, packs, replay, enterprise fleet management, performance tuning, and distributed processing patterns.
Advanced Splunk ingestion experience, including Universal/Heavy Forwarders, HEC, syslog, deployment management, distributed collection, and indexer/data-flow architecture.
Experience implementing Elastic Common Schema (ECS), Splunk Common Information Model (CIM), Open Cybersecurity Schema Framework (OCSF), or similar normalization standards.
Experience with Kafka, Amazon MSK, Kinesis, or other enterprise event-streaming/message-broker technologies.
Experience designing telemetry pipelines across globally distributed, classified, bandwidth-constrained, disconnected, degraded, or intermittently connected environments.
Experience performing telemetry capacity planning using events/bytes per second, ingestion rate, retention, compression, network utilization, storage, and compute measurements.
Experience optimizing SIEM or observability ingestion to reduce licensing, compute, storage, network, or cloud costs while preserving security and mission value.
Experience with containerized or Kubernetes-based data processing, Amazon EKS, OpenTelemetry, Prometheus, Grafana, or comparable cloud-native observability technologies.
Advanced AWS networking and security experience, including IAM, VPC design, private connectivity, load balancing, KMS/encryption, logging, and monitoring.
Experience building automated schema validation, pipeline regression testing, end-to-end telemetry validation, GitOps, or automated configuration promotion.
Experience integrating Cisco/network infrastructure, endpoint security products, vulnerability-management platforms, or other cyber telemetry into centralized analytics platforms.
Experience supporting Zero Trust Architecture, Continuous Diagnostics and Mitigation (CDM), continuous monitoring, RMF, STIG, ATO, or enterprise cybersecurity analytics.
Relevant technical certifications from AWS, Elastic, Splunk, Cribl, HashiCorp, Red Hat, Kubernetes, or cybersecurity certification programs.

Responsibilities

Serve as the senior technical owner and subject-matter expert for AF IC enterprise telemetry collection, ingestion, transformation, normalization, routing, and delivery architectures.
Architect and optimize end-to-end data flows from distributed endpoints, servers, applications, network devices, security platforms, and cloud services into enterprise Elastic, Splunk, and other authorized analytics platforms.
Assess collection architectures for bottlenecks, duplicate or unnecessary data, inefficient routing, bandwidth consumption, data-loss risk, processing overhead, and modernization opportunities.
Engineer and troubleshoot Cribl Stream and Cribl Edge pipelines, including sources, destinations, routes, transformations, parsing, filtering, enrichment, normalization, buffering, and replay.
Engineer Elastic collection and ingestion capabilities using Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, and related technologies.
Support Splunk ingestion and integration using forwarders, syslog, HTTP Event Collector, APIs, intermediate collection tiers, and other approved mechanisms.
Develop data schemas, field mappings, metadata standards, tagging, enrichment, and normalization practices that improve interoperability and downstream usability.
Design resilient pipeline patterns using buffering, queueing, retry, backpressure management, store-and-forward, failover, and recovery for geographically distributed and bandwidth-constrained environments.
Establish pipeline observability and data-quality controls to identify dropped, delayed, malformed, duplicated, or incorrectly routed events and measure throughput, latency, queue depth, and error rates.
Perform capacity and performance analysis across collectors, transport links, processing tiers, AWS services, and downstream analytics platforms; recommend changes that improve efficiency and scale.
Optimize data volume and routing to reduce unnecessary network, compute, storage, cloud, and SIEM ingestion cost without compromising mission or cybersecurity requirements.
Design and support AWS-based ingestion, streaming, processing, storage, and monitoring architectures appropriate to mission requirements.
Develop Python tooling and automation for data transformation, pipeline validation, API integration, configuration generation, health monitoring, testing, and troubleshooting.
Develop and maintain Infrastructure as Code and configuration automation using Terraform, CloudFormation, Ansible, or equivalent technologies.
Manage pipeline code, configuration, and infrastructure artifacts in Git and integrate changes into CI/CD and DevSecOps workflows with automated validation and controlled promotion.
Work with network, cloud, cybersecurity, systems, Elastic, Splunk, and DevSecOps teams to resolve cross-domain data-flow and infrastructure issues.
Ensure data transport and pipeline implementations meet applicable security, access-control, encryption, certificate, auditing, retention, and compliance requirements.
Lead root-cause analysis for data loss, latency, parsing, routing, ingestion, scaling, capacity, and integration failures and implement corrective actions that reduce recurrence.
Develop architecture and data-flow diagrams, implementation plans, engineering standards, runbooks, SOPs, and technical recommendations.
Provide technical leadership for telemetry modernization, platform integration, technology evaluation, migration, and enterprise scaling efforts.