Principal Site Reliability Engineer (SRE)

Ally Bank•Charlotte, NC

About The Position

The Principal Site Reliability Engineer designs and implements highly available, scalable infrastructure for mission-critical production services. The work includes automated deployment pipelines, observability platforms, and disaster recovery, as well as improving reliability across engineering teams. The role advances Ally’s Generative AI and Machine Learning capabilities in reporting and supports product and technology PMO needs with innovative solutions. The engineer leads incident response and postmortem processes, investigates failures in complex distributed systems, and drives systemic reliability improvements. They develop service level objectives (SLOs) and error budgets, applying data to balance feature velocity and reliability and inform organizational decisions. The role also builds tooling and automation to reduce toil and help teams deploy and operate services safely with minimal manual intervention. Ally identifies roles as fully on-site, hybrid, or fully remote; the specific work requirements for this role will be discussed during hiring and may change based on business needs. Ally will not sponsor a new applicant for employment authorization for this position.

Requirements

7+ years of relevant experience.
Bachelor's degree in a relevant field of study or equivalent.
Strong programming skills in languages such as Python, Go, or Node, with the ability to write production-quality code for automation, tooling, and system integration.
Strong incident management and troubleshooting skills, including the ability to diagnose and resolve complex production issues under pressure.
Excellent communication and collaboration skills, including the ability to influence technical direction across multiple teams and mentor engineers at various levels.

Nice To Haves

5+ years of experience in site reliability engineering, systems engineering, or DevOps roles, with a proven track record of maintaining large-scale production systems.
Deep expertise in AWS cloud and infrastructure-as-code tools such as Terraform, CloudFormation, or Pulumi.
Experience defining and measuring SLIs, SLOs, and error budgets and using them to drive reliability improvements and inform product decisions.
Proficiency in AI development.
Extensive experience with container orchestration (ECS or similar) and microservices architectures in production environments.
Proficiency with observability and monitoring tools such as Dynatrace, Prometheus, Grafana, Datadog, New Relic, or similar, and experience building comprehensive monitoring and alerting systems.
Solid understanding of networking concepts, load balancing, CDNs, DNS, and distributed-systems principles, including consensus algorithms and failure modes.
Hands-on experience with CI/CD pipelines and GitOps workflows using tools such as Jenkins, GitHub Actions, ArgoCD, or CircleCI.

Responsibilities

Design and implement highly available, scalable infrastructure systems supporting mission-critical production services, including automated deployment pipelines, observability platforms, and disaster recovery.
Lead incident response and postmortem processes; investigate complex distributed-systems failures, identify root causes, and drive systemic reliability improvements across engineering teams.
Develop and maintain service level objectives (SLOs) and error budgets, using data to balance feature velocity with reliability and guide organizational decision-making.
Build tooling and automation to eliminate toil, improve operational efficiency, and enable teams to deploy and operate services safely with minimal manual intervention.

Benefits

Market-competitive base pay and pay-for-performance annual incentive compensation.
20 paid time off days, 11 paid holidays, and 8 hours of volunteer time off yearly; time off is prorated based on start date and varies by full- or part-time status and management level.
401(k) retirement savings plan with matching and company contributions.
Student loan pay-down and 529 educational savings assistance programs.
Tuition reimbursement.
Employee stock purchase plan.
Financial learning center and financial coach access.
Flexible medical, dental, and vision insurance options.
Life insurance and short- and long-term disability coverage.
Pre-tax Health Savings Account with employer contributions and Healthcare FSA.
Critical illness, accident, and hospital indemnity insurance.
Total well-being program.
Adoption, surrogacy, and fertility assistance.
Paid parental and caregiver leave.
Dependent Day Care FSA, backup child and adult/elder care days, and childcare discounts.
Employee Assistance Program.
Subsidized and discounted Weight Watchers program and other employee discount programs.
Depending on the role, travel allowances, relocation assistance, a signing bonus, and/or equity may be available.