Consulting/Principal Site Reliability Engineer

LexisNexis Reed Tech•Dayton, OH

3d•$118,300 - $219,800

About The Position

The Consulting/Principal Site Reliability Engineer will guide the observability and reliability initiative for Government Systems running on Azure Government. This is an advanced individual-contributor role focused on leading an organizational transition to SLO-based reliability engineering and building development-team buy-in. It is a hands-on SRE position, not a software development role: the engineer will work with other SREs and advise development teams on telemetry and instrumentation strategies rather than write application code. The role serves as the primary technical expert on Datadog and observability practices, helping define meaningful SLOs and error budgets and supporting data-driven reliability decisions. It also includes training and mentoring junior SREs while working within priorities established by the SRE Manager. The position requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance. The technology environment includes Microsoft Azure Government, Linux and Windows servers, Azure DevOps Pipelines, GitHub Actions, Terraform, Datadog, and GitHub Copilot.

Requirements

Significant hands-on experience with Datadog, including dashboards, alerting, SLO and error-budget tracking, and telemetry pipelines.
Proven experience leading an organization's transition to SLO-based reliability engineering.
A track record of guiding development teams on instrumentation and telemetry strategy in an advisory capacity rather than through hands-on application coding.
Deep technical understanding of full-stack observability techniques and the ability to clarify complex incidents or performance issues.
Ability to build templated observability dashboards and configuration as code for others to implement.
Ability to set and influence appropriate organization-wide SLOs and error budgets.
Comfort operating across Linux and Windows server environments.
Experience preparing and assessing SREs for on-call readiness, mentoring them, and identifying training needs.
Skill in facilitating safe, learning-focused post-mortems.
Ability to act as a senior on-call escalation point and guide restoration efforts.
Experience partnering with SRE and Development leads to identify and resolve causes of excessive alerting and pager load.
Advanced understanding of systems design, including high availability, deployment and recovery techniques, failure modes, and system behaviors.
Specialized depth in at least one area such as denial-of-service protection or containerization, with the ability to consult others.
Ability to make resilience and recoverability decisions based on load-testing data and fault-injection results.
Experience documenting SRE design practices and contributing to a broader SRE framework.
Experience contributing to disaster recovery plans, including recovery priorities and procedures, and guiding others through runbooks.
Ability to design and lead disaster recovery exercises and incident-response scenarios.
Experience planning chaos-engineering scenarios involving faults across memory, network, I/O, or distributed systems.
Ability to build engineering and architecture consensus for standard components and services, including platforms and CI/CD.
Experience promoting adoption and standardization and contributing improvements through inner sourcing.
Ability to champion Paved Road adoption and improve efficiency by reducing rework and silos.
Ability to create and share SRE best practices and training materials, coach other SREs, identify development gaps, and eliminate large-scale toil.
U.S. citizenship and ability to obtain and maintain a Public Trust clearance.
Behavioral competencies: collaboration and teamwork; customer and external focus; problem solving and issue analysis; learning agility; relationship building; and developing others.

Nice To Haves

Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling.
Interest in defining best practices for AI-assisted development in an SRE context.

Responsibilities

Guide the observability and reliability initiative and help set its technical direction.
Lead the organization's transition to SLO-based reliability engineering.
Advise development teams on application telemetry and instrumentation strategies without performing application development work.
Serve as the primary technical authority on Datadog, including dashboards, alerting, and SLO tracking.
Recommend service level objectives in partnership with Product and Development teams.
Act as a senior escalation point during incidents and guide restoration efforts.
Improve and evolve the broader SRE framework.
Build disaster recovery plans, including advanced fault-injection scenarios.
Prepare and assess SREs for on-call readiness, mentor them, and identify training needs.
Run psychologically safe post-mortems that foster learning from failure.
Partner with SRE and Development leads to identify and address causes of excessive alerting and pager load.
Make resilience and recoverability decisions informed by load-testing data and fault-injection results.
Document SRE design practices and contribute to the broader SRE framework.
Design and lead disaster recovery practice scenarios and incident-response exercises, identifying knowledge gaps and improvements.
Plan chaos-engineering scenarios that simulate faults across memory, network, I/O, or distributed systems.
Build engineering and architecture consensus for standard components and services on the Paved Road, including platforms and CI/CD.
Promote adoption and standardization, contribute improvements through inner sourcing, and reduce rework and silos.
Create and share SRE best practices and training materials; advise on SRE training curriculum and content.
Coach other SREs, identify development gaps, and help eliminate large-scale toil.