Principal Site Reliability Engineer (SRE)
About The Position
The Principal Site Reliability Engineer will advance Ally’s Generative AI and Machine Learning capabilities in reporting and help meet product and technology PMO needs with innovative solutions. The role designs and implements highly available, scalable infrastructure for mission-critical production services, including deployment pipelines, observability platforms, and disaster recovery. It also leads incident response and postmortems, develops service level objectives and error budgets, and builds automation to reduce operational toil and help engineering teams deploy and operate services safely. Ally describes its work environment as evolving and collaborative, with a focus on customer-oriented technology solutions, work-life balance, teammate well-being, and diversity and inclusion. Work arrangements may be fully onsite, hybrid, or fully remote; the specific requirements for this role will be discussed during hiring and may change based on business needs. Ally will not sponsor a new applicant for employment authorization for this position.
Requirements
- At least 7 years of relevant experience.
- Bachelor’s degree in a relevant field of study or equivalent.
- Strong programming skills in languages such as Python, Go, or Node, including the ability to write production-quality code for automation, tooling, and system integration.
- Strong incident management and troubleshooting skills, including the ability to diagnose and resolve complex production issues under pressure.
- Excellent communication and collaboration skills, with the ability to influence technical direction across teams and mentor engineers at various levels.
Nice To Haves
- At least 5 years of experience in site reliability engineering, systems engineering, or DevOps roles, with a proven track record maintaining large-scale production systems.
- Deep expertise in AWS cloud and infrastructure-as-code tools such as Terraform, CloudFormation, or Pulumi.
- Experience defining and measuring SLIs, SLOs, and error budgets and using them to drive reliability improvements and inform product decisions.
- Proficiency in AI development.
- Extensive production experience with container orchestration such as ECS and microservices architectures.
- Proficiency with observability and monitoring tools such as Dynatrace, Prometheus, Grafana, Datadog, or New Relic, and experience building comprehensive monitoring and alerting systems.
- Solid understanding of networking concepts, load balancing, CDNs, DNS, and distributed systems principles, including consensus algorithms and failure modes.
- Hands-on experience with CI/CD pipelines and GitOps workflows using tools such as Jenkins, GitHub Actions, ArgoCD, or CircleCI.
Responsibilities
- Design and implement highly available, scalable infrastructure systems for mission-critical production services, including automated deployment pipelines, observability platforms, and disaster recovery.
- Lead incident response and postmortem processes, investigate failures in complex distributed systems, identify root causes, and drive systemic reliability improvements across engineering teams.
- Develop and maintain service level objectives (SLOs) and error budgets, using data to balance feature velocity with reliability and guide organizational decisions.
- Build tooling and automation to eliminate toil, improve operational efficiency, and enable teams to deploy and operate services safely with minimal manual intervention.
Benefits
- Market-competitive base pay and performance-based annual incentive plan.
- At least 20 paid time off days at program start, 11 paid holidays, and 8 hours of annual volunteer time off; amounts are prorated based on start date and vary by employment status and management level.
- 401(k) retirement savings plan with matching and company contributions.
- Student loan paydown and 529 educational savings assistance programs.
- Tuition reimbursement.
- Employee stock purchase plan.
- Financial learning center and financial coach access.
- Medical, dental, and vision insurance options.
- Employee, spouse, and child life insurance.
- Short- and long-term disability coverage.
- Health Savings Account with employer contributions and Healthcare FSA.
- Critical illness, accident, and hospital indemnity insurance.
- Total well-being program.
- Adoption, surrogacy, and fertility assistance.
- Paid parental and caregiver leave.
- Dependent Day Care FSA, backup child and adult/elder care days, and childcare discounts.
- Employee assistance program.
- Subsidized and discounted Weight Watchers program and other employee discount programs.
- Depending on the role, travel allowances, relocation assistance, signing bonus, and/or equity may be available.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
