Senior Observability Platform Engineer - Alerting

CACI•Reston, VA

10d

About The Position

The Senior Observability Platform Engineer—Grafana, Alerting, and SRE Focus turns telemetry into actionable dashboards, alerts, service-level objectives, incident workflows, and reliable operational outcomes. The role combines telemetry, alert engineering, platform reliability engineering, incident-management support, and observability user enablement. It enables platform operators, engineers, security teams, leadership, and FinOps stakeholders to use data effectively through governed dashboards, actionable alerts, and repeatable operational processes. The engineer deploys and maintains Grafana as the primary visualization and alerting interface; builds role-based dashboards; designs alerting and incident workflows; establishes service health and reliability practices; writes and optimizes queries; helps application teams develop operational views; and works with IAM, security, and data-governance stakeholders to apply appropriate access controls and data-handling requirements.

Requirements

Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
7+ years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
3+ years of advanced Grafana administration, dashboard development, and alerting experience.
Strong experience defining and operating SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement processes.
Demonstrated ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
Proficiency in PromQL and/or LogQL.
Experience integrating alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.
Ability to meet the minimum clearance requirement of TS/SCI with Polygraph.

Nice To Haves

Experience with Elastic Query DSL/KQL.
Experience with Grafana Enterprise, including advanced RBAC, reporting, incident-management features, and offline licensing in disconnected environments.
Experience with Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry, and Grafana Alloy.
Experience building dashboards for security operations, cloud cost management, OpenCost, or executive operational reporting.
Experience in air-gapped, classified, DoD, federal, or other highly regulated environments.
Familiarity with NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence, and FIPS constraints.

Responsibilities

Deploy, administer, secure, and maintain Grafana as the primary visualization and alerting interface.
Build and maintain role-based dashboards for platform engineering, SRE, security operations, application teams, FinOps, and leadership.
Design, implement, tune, and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage issues, and SLO violations.
Establish alert standards and practices for severity definitions, deduplication, grouping, inhibition, routing, escalation, silencing, suppression, and maintenance windows.
Define, implement, and maintain SLIs, SLOs, error budgets, service-health standards, and reliability reporting for services.
Build and maintain meta-monitoring dashboards covering pipeline throughput, dropped samples, query latency, ingestion rate, storage utilization, node health, alerting status, and identity-provider health.
Write and optimize PromQL, LogQL, TraceQL, Elastic Query DSL/KQL, or other query languages relevant to the approved platform architecture.
Partner with application teams to build RED-method dashboards, service-level monitoring, trace-to-log correlation, and deployment-aware operational views.
Work with IAM, security, and data-governance stakeholders to implement dashboard, query, and administrative access controls.
Ensure sensitive telemetry is displayed and accessed in accordance with approved data-handling and least-privilege requirements.