Observability Pipeline Engineer

Booz Allen Hamilton•McLean, VA

1h•$86,800 - $198,000•Hybrid

About The Position

Are you looking for an opportunity to combine deep technical skill with big picture thinking to make an impact on the systems people depend on every day? You understand that observability starts with the data, and that every dashboard, alert, and investigation is only as good as the telemetry underneath it. As an observability pipeline engineer on our team, you'll own the telemetry that feeds logging, monitoring, and security analytics across commercial and accredited government environments. Your customers will trust you not only to build and operate the pipeline in Cribl and Splunk, but to evolve it — delivering source onboarding as code, applying AI to the parser and schema work that has traditionally consumed days, and building a pipeline that detects its own failures before anyone downstream is affected. On our team, you'll broaden your skills in OpenTelemetry and telemetry standards. This role builds the pipeline itself. The work sits upstream of the dashboards, alerts, and searches everyone else uses, and its quality determines theirs. The ideal candidate comes from a data engineering, log platform, SIEM, or observability infrastructure background and has run high-volume pipelines in production under real cost and retention constraints.

Requirements

4+ years of experience engineering telemetry, log, or event data pipelines at enterprise scale
Experience building and operating a telemetry routing tier such as Cribl Stream, OpenTelemetry Collector, Vector, Fluent Bit, or Logstash, including schema design, parsing, normalization, reduction, and retention tiering
Experience with a query language, including SPL, DQL, LogQL, PromQL, or KQL
Experience managing data volume, cardinality, and cost under volume-based or consumption-based licensing
Experience managing configuration as code, including infrastructure as code such as GitOps practice and Terraform, across multiple environments
Experience writing production code in a language such as Python, working against REST APIs, and administering Linux and Windows systems
Ability to learn new query languages
Secret clearance
Bachelor's degree in CS, Information Systems, or Engineering

Nice To Haves

Experience designing OpenTelemetry Collector processors and pipelines at scale
Experience handling PII or sensitive data in telemetry, including masking and in-flight data-protection controls
Experience applying AI tooling to data engineering work, including parser generation, schema inference, or anomaly detection on data flows and validating the output before trusting it
Experience with DoD cloud environments such as Azure or AWS IL5/IL6, including change control and continuous monitoring obligations
Experience with CIM-compliant data engineering
Cribl Certified Observability Engineer (CCOE) or equivalent certification
Splunk Enterprise Certified Architect Certification

Responsibilities

Build and evolve the telemetry pipeline that carries logs, metrics, and events across every accredited environment, including sources, parsing, enrichment, normalization, routing, and destinations.
Engineer the Splunk log platform from distributed architecture and index design through data onboarding and search performance.
Automate source onboarding end to end so that adding a data source is a pull request rather than a ticket, with all pipeline configuration managed as code under GitOps.
Apply AI tooling to parser generation, schema mapping, and normalization, collapsing work that used to take days.
Instrument the pipeline to detect its own failures, including silent sources, schema drift, and volume and cost anomalies, before anyone downstream notices.
Design reduction, cardinality, retention, and masking strategies that control cost, protect sensitive data by destination, and keep the operational record provable.