Consulting/Principal Site Reliability Engineer

LexisNexis Reed Tech•Dayton, PA

About The Position

The Consulting/Principal Site Reliability Engineer will guide the observability and reliability initiative for Government Systems running on Azure Government. This is an advanced, hands-on individual-contributor SRE role focused on leading an organizational transition to SLO-based reliability engineering and securing development-team adoption. The role works alongside other SREs and advises development teams on telemetry and instrumentation strategy; it does not involve writing application code. The engineer serves as the primary technical authority on Datadog and observability practices, helps define meaningful SLOs and error budgets, and supports data-driven reliability decisions. The role also includes mentoring junior SREs and working within priorities set by the SRE Manager. The environment includes Microsoft Azure Government, Linux and Windows servers, Azure DevOps Pipelines, GitHub Actions, Terraform, Datadog, and GitHub Copilot. The position requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance. The U.S. national base pay range is $118,300–$219,800; geographic differentials may apply. The position is eligible for an annual incentive bonus. Country-specific benefits are available, but specific benefits are not listed in the posting.

Requirements

Significant hands-on experience with Datadog, including dashboards, alerting, SLO and error-budget tracking, and telemetry pipelines.
Proven experience leading an organization's transition to SLO-based reliability engineering.
Track record of advising development teams on instrumentation and telemetry strategy rather than writing application code.
Deep technical understanding of full-stack observability techniques and the ability to clarify complex incidents or performance issues.
Ability to build templated observability dashboards and configuration as code.
Ability to set and influence appropriate SLOs and error budgets across an organization.
Comfort operating across Linux and Windows server environments.
Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs.
Skill in facilitating blameless post-mortems that foster learning from failure.
Comfort serving as a senior on-call escalation point and guiding service restoration.
Experience partnering with SRE and Development leads to identify and fix causes of excessive alerting and pager load.
Advanced understanding of systems design, including high availability, deployment and recovery techniques, failure modes, and system behaviors.
Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to consult others.
Ability to make resilience and recoverability decisions based on load-testing data and fault-injection results.
Experience contributing to disaster recovery plans and guiding others through runbooks.
Experience designing and leading disaster recovery practice scenarios and incident response exercises.
Experience planning chaos engineering scenarios across memory, network, I/O, or distributed systems.
Ability to build engineering and architecture consensus for standard components and services, including platforms and CI/CD.
Ability to promote adoption and standardization, including contributions through inner sourcing.
Ability to create and share SRE best practices and training materials, coach other SREs, identify development gaps, and eliminate large-scale toil.
U.S. citizenship and ability to obtain and maintain a Public Trust clearance.

Nice To Haves

Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling.
Interest in helping define best practices for AI-assisted development in an SRE context.
Specialized depth in denial-of-service protection or containerization.

Responsibilities

Guide the observability and reliability initiative and work with other SREs to set technical direction for measuring and improving reliability.
Lead the transition to SLO-based reliability engineering, drawing on prior experience leading such a transition.
Advise development teams on application telemetry and instrumentation strategy without performing application development work.
Serve as the primary technical authority on Datadog, including dashboarding, alerting, and SLO tracking.
Recommend service level objectives in partnership with Product and Development teams.
Act as a senior escalation point during incidents and guide restoration efforts.
Improve and evolve the broader SRE framework, including documenting SRE design practices.
Build disaster recovery plans, including recovery priorities, procedures, runbooks, and advanced fault-injection scenarios.
Design and lead disaster recovery practice scenarios and incident response exercises, identifying knowledge gaps and improvements.
Plan chaos engineering scenarios that simulate faults across memory, network, I/O, or distributed systems.
Build templated observability dashboards and configuration as code so teams can implement them efficiently.
Partner with SRE and Development leads to identify and address root causes of excessive alerting and pager load.
Build engineering and architecture consensus for standard components and services on the Paved Road, including platforms and CI/CD.
Promote Paved Road adoption, standardization, and inner sourcing to reduce rework and silos.
Advise on SRE training curriculum and content, mentor and coach other SREs, and identify training needs and development gaps.
Identify and eliminate large-scale toil.
Work within priorities established by the SRE Manager.