Senior DevOps Engineer, Observability and FinOps

CACI • Reston, VA

7d

About The Position

This engineer makes mission systems visible and their costs accountable on networks that cannot connect to the internet. The role combines platform engineering—including Kubernetes/OpenShift, infrastructure as code, and CI/CD—with full-stack development to build dashboards, APIs, and internal tools for operators and financial stakeholders.

The engineer deploys and operates observability and cost-management capabilities across AWS, OpenShift, and on-premises resources, including in air-gapped environments. The work includes instrumenting applications and infrastructure, building operational and cost dashboards, optimizing resource usage, automating deployment and delivery, developing integrations and lightweight applications, and documenting and training users. The position requires an active TS/SCI with polygraph clearance and a relevant bachelor's degree.

Requirements

• Active TS/SCI with polygraph clearance.
• Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
• At least 3 years deploying and operating at least one enterprise stack: Grafana/Prometheus ecosystem, Elastic (ELK), Dynatrace, or Splunk.
• Experience with metrics, logs, and traces instrumentation and working knowledge of OpenTelemetry or Prometheus exporters.
• Experience building operational dashboards and alert rules used by operators; understanding of SLI/SLO concepts.
• At least 3 years of production Kubernetes experience; Red Hat OpenShift is strongly aligned.
• At least 2 years with AWS, including EC2, S3, EKS, IAM, CloudWatch, and Cost and Usage Reports.
• Demonstrated experience analyzing cloud spend, applying tagging strategies, and delivering at least one cost optimization outcome.
• Experience with Terraform and Ansible, and Helm charts or Operators.
• Experience building and maintaining pipelines in GitLab CI, Jenkins, or an equivalent, and familiarity with Git workflows.
• Proficiency in Python or Go and scripting with Bash; ability to write and query SQL and PromQL or equivalent.
• RHEL administration and troubleshooting experience.
• Experience hardening systems to DISA STIGs; familiarity with RMF and NIST SP 800-53 controls.
• Experience working in air-gapped or disconnected networks, including offline package and image management.

Nice To Haves

• Hands-on experience with Grafana Alloy, Mimir, Loki, and Tempo at scale.
• Production experience with two or more of Grafana, Elastic, and Dynatrace, enabling side-by-side judgment.
• Experience with store-and-forward pipelines, edge collectors, Kafka, or similar buffering over intermittent links.
• Experience moving data or software through cross-domain solutions (CDS) or approved transfer processes.
• Experience with OpenCost or Kubecost, CloudHealth, Apptio Cloudability, or AWS Cost Explorer and Cost and Usage Report analytics.
• Experience with Istio or a comparable service mesh, including mesh-level metrics and tracing.
• Experience with cluster monitoring and user workload monitoring, Operators, OpenShift GitOps, and OpenShift Pipelines.

Responsibilities

• Deploy, configure, and operate observability stacks on each network, using tools such as Grafana Alloy, Mimir, Loki, Tempo, Elastic, or Dynatrace.
• Instrument applications and infrastructure with OpenTelemetry and Prometheus exporters, and standardize labels, tags, and naming.
• Build real-time mission status dashboards and service health views for operators, engineers, and leadership.
• Implement cost visibility and allocation across AWS, OpenShift, and on-premises resources using tagging standards and showback or chargeback models.
• Build cost and utilization dashboards that join billing exports with observability metrics, including unit cost by mission application, namespace, and program.
• Identify and execute cost optimizations, including rightsizing, idle resource cleanup, storage lifecycle policies, and Reserved Instance and Savings Plan coverage analysis.
• Automate infrastructure and stack deployment with Terraform, Ansible, and Helm or Operators, and manage configuration through GitOps.
• Build and maintain CI/CD pipelines, including image scanning, signing, and promotion into air-gapped registries.
• Package software and dependencies for offline transfer and cross-domain delivery.
• Develop lightweight web apps, APIs, and Grafana plugins or panels for custom status views and cost reports.
• Write data integration scripts in Python or Go to pull from billing exports, CMDBs, and ticketing systems.
• Document runbooks, architecture diagrams, and user guides; train operators and mentor junior engineers.

Benefits

• Flexible time off benefit.
• Learning resources and learning and development opportunities.
• Healthcare benefits.
• Wellness benefits.
• Financial benefits.
• Retirement benefits.
• Family support benefits.
• Continuing education benefits.
• Time off benefits.
• Competitive compensation and benefits options designed to support employees and their families.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior