Senior Observability Engineer - Platform and Telemetry

CACI International•Rome, NY
•$90,300 - $189,600

About The Position

The Senior Observability Engineer designs, deploys, operates, and improves the core observability platform, owning the technical foundation for secure telemetry collection, processing, storage, retention, resiliency, and platform self-observability across classified and disconnected OpenShift environments. The role supports the approved telemetry architecture, including Elastic and Grafana, Prometheus, Loki, and S3-compatible object storage. The engineer deploys and operates the telemetry platform on Red Hat OpenShift; builds highly available ingestion, processing, storage, retention, backup, and recovery capabilities; and designs telemetry collection pipelines for infrastructure, Kubernetes/OpenShift, applications, security, cloud, CI/CD, and cost data. The position also manages telemetry storage and lifecycle procedures, instruments the platform to monitor its own health and performance, and collaborates with technical and security partners to ensure reliable telemetry availability and data preparation for approved AI capabilities.

Requirements

  • Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
  • 7+ years of experience in platform engineering or observability engineering.
  • 3+ years of hands-on production experience with Kubernetes or Red Hat OpenShift.
  • Hands-on experience with Elastic Stack, including Elasticsearch, ingest pipelines, index lifecycle management, storage tiers, and retention; and/or Grafana LGTM components including Alloy, Prometheus/Mimir, Loki, Tempo, and object storage.
  • Experience with OpenTelemetry, OTLP, metrics, logs, traces, structured events, and telemetry data processing.
  • Strong experience with Linux, containers, Helm, Kustomize, Git, GitLab CI/CD, Argo CD or OpenShift GitOps, and infrastructure-as-code.
  • Experience with storage lifecycle management, backup and recovery, capacity planning, performance tuning, and distributed-system troubleshooting.
  • Ability to automate with Bash, Python, Go, or a comparable language.
  • Minimum active clearance required to start: TS/SCI with Polygraph.

Nice To Haves

  • Experience operating systems in air-gapped, classified, disconnected, or highly regulated environments.
  • Experience with NIST SP 800-53, NIST SP 800-137, FIPS requirements, RMF/ATO processes, and secure software-supply-chain practices.
  • OpenShift, Kubernetes, Grafana, Elastic, Prometheus, or OpenTelemetry certifications.
  • Experience with S3-compatible object storage, OpenCost, FinOps telemetry, chargeback, or showback systems.
  • Familiarity with AWS, Azure, hybrid-cloud telemetry, and enterprise identity/security tooling.

Responsibilities

  • Deploy, configure, secure, upgrade, and operate the telemetry platform on Red Hat OpenShift.
  • Build and maintain highly available telemetry ingestion, processing, storage, retention, backup, and recovery capabilities.
  • Design and operate collection pipelines for infrastructure, Kubernetes/OpenShift, application, security, cloud, CI/CD, and cost telemetry.
  • Configure and maintain Grafana, OpenTelemetry Collectors, OTLP endpoints, exporters, receivers, processors, and/or Elastic ingest pipelines, as applicable.
  • Configure and manage metrics, logs, traces, and archival storage back ends.
  • Implement data retention, compaction, lifecycle management, backup, restoration, and disaster-recovery procedures.
  • Implement telemetry and dashboards for the platform’s own health, including pipeline throughput, dropped samples, query latency, ingestion rate, storage utilization, component health, and resource consumption.
  • Instrument pipeline components to capture throughput, queue depth, backpressure, dropped-record reasons, CPU and memory utilization, and end-to-end pipeline latency.
  • Partner with the Observability Architect, OpenShift DevSecOps Engineer, Security Engineer/ISSO, Data Engineer, and Grafana/SRE-focused engineer.
  • Support reliable telemetry availability and data preparation for anomaly detection and other approved AI capabilities.

Benefits

  • Flexible time off benefit.
  • Learning resources.
  • Healthcare benefits.
  • Wellness benefits.
  • Financial and retirement benefits.
  • Family support benefits.
  • Continuing education benefits.
  • Time off benefits.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service