Senior Site Reliability Engineer

Peraton • ,

18h • $104,000 - $166,000

About The Position

Peraton Labs is hiring a Senior Site Reliability Engineer to own the end-to-end reliability of an enterprise application. This is an engineering-focused role involving production Python and Rust development as well as reliability practices. The engineer will define service-level indicators and objectives, manage error budgets, build monitoring and alerting, and improve services through instrumentation, health checks, resilient design, and production code changes. The role also covers Kubernetes and AWS operations, infrastructure as code, CI/CD, incident management, security and compliance, load testing, capacity planning, and mentoring engineers.

Requirements

• Bachelor’s degree and 8+ years of experience in SRE, platform, or software engineering roles with production reliability ownership.
• Strong software engineering skills in Python; ability to read, debug, and ship production application code, including performance, concurrency, and failure-handling fixes.
• Experience defining SLIs and SLO targets with product and engineering, operating error-budget policies, and using burn rates to inform release and prioritization decisions.
• Experience owning monitoring and alerting with CloudWatch, Prometheus/Grafana, or equivalent, including actionable low-noise alerts tied to SLOs and multi-window burn rates, alert routing, and on-call hygiene.
• Experience designing service, SLO, and executive-level dashboards in Grafana or equivalent.
• Experience instrumenting Python and Rust services with metrics, structured logging, and distributed tracing using OpenTelemetry or equivalent.
• Experience building application health checks and scripted synthetic user-journey checks that exercise real authentication flows and critical transactions.
• Experience leading incident response, conducting blameless postmortems, and driving corrective actions to completion.
• Strong Amazon EKS/Kubernetes operations experience, including Helm, RBAC, pod identity, GitOps, upgrades, and troubleshooting.
• Solid AWS experience across core services such as VPC, IAM, S3, RDS, CloudTrail, CloudWatch, and KMS.
• CI/CD experience with GitHub Actions and/or GitLab CI, including reliability and performance gates.
• Experience with infrastructure as code using OpenTofu or Terraform.
• Experience with Agile delivery practices, including trunk-based development, MR-driven change, and automated quality gates.
• Security and compliance awareness, including secrets management, least privilege, and audit evidence.
• U.S. citizenship is required.

Nice To Haves

• Production Rust experience, such as async/Tokio or FFI with Python via PyO3.
• Load and performance testing experience with k6, Locust, or equivalent, and capacity planning.
• Experience with chaos engineering and fault-injection testing.
• Experience with progressive delivery, including canary or blue-green deployments with automated rollback based on SLOs or health-check failures.
• Experience with Big Bang (Platform One), IronBank, or hardened-image pipelines.
• Prior work on systems pursuing or holding an ATO.
• Contributions to observability tooling or Rust/Python open-source projects.

Responsibilities

• Own end-to-end reliability of the enterprise application; define SLIs and SLOs with product and engineering, enforce error-budget policy, and use burn rates to guide release and prioritization decisions.
• Write, review, and ship production Python and Rust code to address performance, concurrency, and failure-handling issues and improve observability and resilience.
• Instrument services with metrics, structured logging, and distributed tracing using OpenTelemetry or an equivalent.
• Own monitoring and alerting, including building SLO-based burn-rate alerts, managing alert routing and on-call hygiene, and designing service, SLO, and executive dashboards.
• Build in-application health checks and synthetic user-journey tests that exercise real authentication flows and critical transactions.
• Lead incident response, run blameless postmortems, and drive corrective actions to completion.
• Operate Amazon EKS and Kubernetes, including Helm, RBAC, pod identity, GitOps, upgrades, and troubleshooting; operate core AWS services managed through OpenTofu or Terraform.
• Maintain GitHub Actions and/or GitLab CI pipelines with reliability and performance gates, supporting trunk-based, MR-driven Agile delivery and progressive rollouts.
• Enforce secrets management and least privilege, and produce audit evidence for NIST 800-171, CMMC, FedRAMP, or DoD/ATO environments.
• Drive load testing, capacity planning, and chaos engineering.
• Mentor engineers on reliability practices.

Benefits

• Medical benefits
• Dental insurance
• Vision insurance
• Life insurance
• Health savings account
• Short- and long-term disability
• Employee assistance program
• Parental leave
• 401(k)
• Paid time off for vacation
• Company-paid holidays
• Potential eligibility for overtime, shift differential, and a discretionary bonus, depending on the position

What This Job Offers

• Job Type: Full-time
• Career Level: Senior