Observability Platform Architect

Booz Allen Hamilton•McLean, VA

9h•$86,800 - $198,000•Hybrid

About The Position

Our team runs observability as a service for the enterprise, built on a platform that tells us whether its business services are healthy, so problems are caught before users notice. The platform supports more than 250 business services across multiple portfolios and environments. In this role, you will own the platform's architecture, instrumentation strategy, and entity model, and build the automation that lets a small team support it at this scale. More and more of that automation is driven by AI agents, so you will also decide what those agents can do on their own and how we confirm they made the right call. This is a hands-on engineering role, not an advisory one. You will set the standards, guide the engineers who build to them, and build alongside them. The ideal candidate comes from a platform engineering, observability, or site reliability engineering background and has owned an observability platform rather than just used one. Due to the nature of work performed within this facility, U.S. citizenship is required.

Requirements

8+ years of engineering experience
3+ years of experience owning an enterprise observability or APM platform, including its architecture, upgrades, standards, cost, and production support
2+ years of experience setting technical direction for other engineers, including architecture decisions, code review, and mentoring
Experience writing and optimizing complex queries against observability data such as DQL or PromQL
Experience replacing recurring manual platform work with production automation, including automated alert correlation, triage, or remediation, and owning whether it stayed correct over time
Experience using AI in operations, including vendor AI assistants, LLM tools, or agents
Experience with infrastructure as code and CI/CD such as Terraform, GitHub Actions, Azure DevOps, or GitLab, and deploying monitoring agents or collectors at scale across cloud, Kubernetes, and hybrid environments
Ability to write, review, and ship code in Python, Go, or JavaScript against REST APIs
Bachelor's degree in CS, Information Systems, or Engineering

Nice To Haves

Experience with Dynatrace in depth, including Grail, DQL, OpenPipeline, Workflows, Site Reliability Guardian, Davis, Dashboards, and the Dynatrace Operator on Kubernetes, with configuration managed through the Terraform provider
Experience with Grafana and its broader stack such as Prometheus, Loki, Tempo, or Mimir and what it costs to run at scale
Experience with ServiceNow Event Management and CMDB, including event rules, alert correlation, and closed-loop incident creation from an observability platform
Experience building LLM-based agents and the tooling they use such as agent frameworks or Model Context Protocol (MCP) servers
Experience with OpenTelemetry, including collectors, instrumentation, semantic conventions, and the practical limits of vendor-neutral instrumentation
Experience operating one platform across more than one environment or security boundary, and managing the drift and duplication that creates
ITIL 5 Foundation or ITIL 4 Practitioner: Monitoring and Event Management Certification

Responsibilities

Translate the service owner's priorities into the platform's technical plan, set technical direction through architecture decisions, design reviews, configuration governance, and mentoring engineers, and guide the team's day-to-day technical work.
Architect and build the observability platform on Dynatrace, Grafana, and ServiceNow Event Management across every environment, including its integrations with ITSM, CMDB, and CI/CD, and keep environments from drifting into separately maintained estates.
Keep the platform healthy, including upgrades, licensed consumption within budget, and escalation for complex platform issues, major incidents, and vendor support.
Drive OpenTelemetry adoption and automate the platform itself, including collector configuration, service onboarding, monitoring agent deployment, tagging, and alert provisioning, delivered as code through CI/CD and tested in non-production before release.
Build and deploy AI agents for detection, triage, and remediation, connected to the platform's APIs and tools, and own the platform's AI capability end to end, including what agents may do unattended, how we verify recovery, and how incorrect actions get caught.
Own signal quality, including coverage, detection accuracy, and false positive rate, along with the entity model, dashboards, and SLOs leadership depends on, and drive down cost per monitored service.