Senior Observability Platform Engineer - Alerting

CACI International • Rome, NY

22h • $90,300 - $189,600

About The Position

The Senior Observability Platform Engineer is responsible for turning telemetry into actionable dashboards, alerts, service-level objectives, incident workflows, and reliable operational outcomes. The role combines telemetry, alert engineering, platform reliability engineering, incident-management support, and observability user enablement. It helps platform operators, engineers, security teams, leadership, and FinOps stakeholders use data effectively through governed dashboards, actionable alerts, and repeatable operational processes.

The engineer deploys, administers, secures, and maintains Grafana as the primary visualization and alerting interface; develops role-based dashboards; designs and operates alerting standards and incident workflows; and defines SLIs, SLOs, error budgets, service-health standards, and reliability reporting. The role also develops meta-monitoring, queries telemetry using languages appropriate to the platform, partners with application teams on operational views, and works with IAM, security, and data-governance stakeholders to ensure access controls and sensitive telemetry handling meet approved requirements.

This is a regular, full-time position requiring TS/SCI with Polygraph clearance. Travel of up to 10% may be required within the Continental US. The proposed national salary range is $90,300–$189,600. CACI describes a culture of integrity and trust, flexible time off, learning resources, and comprehensive benefits covering healthcare, wellness, financial and retirement support, family support, continuing education, and time off.

Requirements

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
• At least 7 years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
• At least 3 years of advanced Grafana administration, dashboard development, and alerting experience.
• Strong experience defining and operating SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement processes.
• Demonstrated ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
• Proficiency in PromQL and/or LogQL.
• Experience integrating alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
• Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
• Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.
• TS/SCI with Polygraph clearance.

Nice To Haves

• Experience with Elastic Query DSL/KQL.
• Experience with Grafana Enterprise, including advanced RBAC, reporting, incident-management features, and offline licensing in disconnected environments.
• Experience with Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry, and Grafana Alloy.
• Experience building dashboards for security operations, cloud cost management, OpenCost, or executive operational reporting.
• Experience in air-gapped, classified, DoD, federal, or other highly regulated environments.
• Familiarity with NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence, and FIPS constraints.

Responsibilities

• Deploy, administer, secure, and maintain Grafana as the primary visualization and alerting interface.
• Build and maintain role-based dashboards for platform engineering, SRE, security operations, application teams, FinOps, and leadership.
• Design, implement, tune, and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage issues, and SLO violations.
• Establish alert standards and practices for severity definitions, deduplication, grouping, inhibition, routing, escalation, silencing, suppression, and maintenance windows.
• Define, implement, and maintain SLIs, SLOs, error budgets, service-health standards, and reliability reporting for onboarded services.
• Build and maintain meta-monitoring dashboards covering pipeline throughput, dropped samples, query latency, ingestion rate, storage utilization, node health, alerting status, and identity-provider health.
• Write and optimize PromQL, LogQL, TraceQL, Elastic Query DSL/KQL, or other relevant query languages based on the approved platform architecture.
• Partner with application teams to build RED-method dashboards, service-level monitoring, trace-to-log correlation, and deployment-aware operational views.
• Work with IAM, security, and data-governance stakeholders to implement appropriate dashboard, query, and administrative access controls.
• Ensure sensitive telemetry is displayed and accessed in accordance with approved data-handling and least-privilege requirements.

Benefits

• Flexible time off benefit.
• Learning resources and learning and development opportunities.
• Comprehensive benefits, including healthcare, wellness, financial, retirement, family support, continuing education, and time off benefits.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior