Senior DevOps Engineer, Observability and FinOps

CACI International•Rome, NY

8d•$98,500 - $206,800

About The Position

The Senior DevOps Engineer, Observability and FinOps makes mission systems visible and their costs accountable on networks that cannot connect to the internet. The role combines platform engineering—including Kubernetes/OpenShift, infrastructure as code, and CI/CD—with full-stack development to deliver dashboards, APIs, and internal tools for operators and financial stakeholders. The engineer deploys and operates observability and cost-management capabilities across disconnected networks, instruments applications and infrastructure, builds mission-status and service-health views, and supports cost allocation and optimization across AWS, OpenShift, and on-premises resources. The role also automates infrastructure and software delivery, develops integrations and custom tools, packages software for offline and cross-domain delivery, and documents and shares operational knowledge.

Requirements

Active TS/SCI clearance with polygraph.
Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
At least 3 years deploying and operating at least one enterprise stack: Grafana/Prometheus ecosystem, Elastic (ELK), Dynatrace, or Splunk.
Experience instrumenting metrics, logs, and traces, with working knowledge of OpenTelemetry or Prometheus exporters.
Experience building operational dashboards and alert rules used by operators; understanding of SLI/SLO concepts.
At least 3 years of production Kubernetes experience; Red Hat OpenShift experience is strongly aligned.
At least 2 years of AWS experience, including EC2, S3, EKS, IAM, CloudWatch, and Cost and Usage Reports.
Demonstrated experience analyzing cloud spend, applying tagging strategies, and delivering at least one cost-optimization outcome.
Experience with Terraform and Ansible, and Helm charts or Operators.
Experience building and maintaining pipelines in GitLab CI, Jenkins, or an equivalent, as well as Git workflows.
Proficiency in Python or Go plus Bash scripting; ability to write and query SQL and PromQL or equivalent.
RHEL administration and troubleshooting experience.
Experience hardening systems to DISA STIGs; familiarity with RMF and NIST SP 800-53 controls.
Experience working in air-gapped or disconnected networks, including offline package and image management.

Nice To Haves

Hands-on experience with Grafana Alloy, Mimir, Loki, and Tempo at scale.
Production experience with two or more of Grafana, Elastic, and Dynatrace, enabling side-by-side evaluation.
Experience with store-and-forward pipelines, edge collectors, and Kafka or similar buffering over intermittent links.
Experience moving data or software through cross-domain solutions (CDS) or approved transfer processes.
Experience with OpenCost or Kubecost, CloudHealth, Apptio Cloudability, or AWS Cost Explorer and Cost and Usage Report analytics.
Experience with Istio or a comparable service mesh, including mesh-level metrics and tracing.
Experience with cluster monitoring and user workload monitoring, Operators, OpenShift GitOps, and Pipelines.

Responsibilities

Deploy, configure, and operate observability stacks on each network, using platforms such as Grafana Alloy, Mimir, Loki, Tempo, Elastic, or Dynatrace.
Instrument applications and infrastructure with OpenTelemetry and Prometheus exporters, and standardize labels, tags, and naming.
Build real-time mission-status dashboards and service-health views for operators, engineers, and leadership.
Implement cost visibility and allocation across AWS, OpenShift, and on-premises resources using tagging standards and showback or chargeback models.
Build cost and utilization dashboards that combine billing exports with observability metrics, including unit cost per mission application, namespace, or program.
Identify and execute cost optimizations such as rightsizing, idle-resource cleanup, storage lifecycle policies, and Reserved Instance and Savings Plan coverage analysis.
Automate infrastructure and stack deployment with Terraform, Ansible, and Helm or Operators; manage configuration through GitOps such as Argo CD.
Build and maintain CI/CD pipelines using GitLab CI, Jenkins, or Tekton, including image scanning, signing, and promotion into air-gapped registries.
Package software and dependencies for offline transfer and cross-domain delivery.
Develop lightweight web apps, APIs, and Grafana plugins or panels for custom status views and cost reports.
Write data integration scripts in Python or Go to pull data from billing exports, CMDBs, and ticketing systems.
Document runbooks, architecture diagrams, and user guides; train operators and mentor junior engineers.