Principal Site Reliability Engineer (Cloud, Observability & Automation)
About The Position
The Enterprise Application Support (EAS) team supports critical applications across DTCC’s ITP and ECS business lines, ensuring the reliability, scalability, and performance of enterprise platforms. As a Principal Site Reliability Engineer, you will lead reliability engineering across mission-critical enterprise platforms and applications, partnering with engineering, product, infrastructure, cloud, and operations teams to improve reliability, resiliency, observability, and operational excellence. This senior technical leadership role establishes reliability standards, advances service health and observability, reduces operational risk, and enables highly available, scalable systems through automation, modern engineering practices, and AI-enabled technologies. DTCC’s Information Technology group delivers secure, reliable technology solutions that support the global capital markets. The role is offered under a flexible hybrid model requiring three days onsite and two days remote; onsite days include Tuesday, Wednesday, and a third day determined by the team or employee. Compensation is competitive and includes base pay and an annual incentive. DTCC also describes comprehensive health and life insurance and well-being benefits, pension/retirement benefits, and paid time off and personal/family care and other leaves of absence.
Requirements
- Bachelor’s degree in computer science, engineering, information technology, or equivalent experience.
- 8–10+ years of experience in Site Reliability Engineering, Production Engineering, DevOps, Application Support Engineering, Software Engineering, or related disciplines.
- Experience supporting large-scale, highly available production environments.
- Deep understanding of Site Reliability Engineering principles and operational excellence practices.
- Strong experience defining and managing SLIs, SLOs, error budgets, and reliability metrics.
- Expertise with observability and monitoring platforms such as Grafana, Splunk, Dynatrace, ITSI, OpenTelemetry, and related technologies.
- Experience with distributed systems, microservices architectures, cloud-native applications, and enterprise platforms.
- Strong programming and automation skills using Python, Java, Go, or similar languages.
- Experience leading major incident management, root cause investigations, and service improvement programs.
- Strong knowledge of Linux/Unix systems, networking fundamentals, performance tuning, and troubleshooting methodologies.
- Experience with AWS and modern cloud technologies.
- Excellent communication, collaboration, and stakeholder management skills.
- Ability to influence engineering teams and drive reliability-focused outcomes across complex organizations.
Nice To Haves
- Experience implementing observability strategies and enterprise monitoring standards.
- Experience with Kubernetes, containers, CI/CD pipelines, Infrastructure as Code, and modern software delivery practices.
- Experience with AI-assisted engineering platforms such as Amazon Q, Kiro, GitHub Copilot, or similar technologies.
- Experience in financial services or other highly regulated, mission-critical environments.
- Proven track record driving reliability transformation initiatives and measurable operational improvements.
Responsibilities
- Drive reliability, scalability, performance, and resiliency improvements across critical enterprise applications and services.
- Define and implement reliability engineering practices, including SLIs, SLOs, error budgets, and operational KPIs.
- Design and implement observability solutions using platforms such as Grafana, Splunk, Dynatrace, OpenTelemetry, and related monitoring technologies.
- Establish monitoring, alerting, telemetry, and service-health frameworks for proactive issue detection and rapid resolution.
- Partner with development teams to incorporate reliability, observability, resiliency, and operational readiness into application design and delivery.
- Perform architecture reviews and identify opportunities to improve availability, performance, fault tolerance, and recoverability.
- Lead major incident response, root cause analysis, and continuous service improvement initiatives.
- Drive alert optimization, incident reduction, and MTTR improvement through data-driven operational analysis.
- Champion resiliency engineering, including failure testing, disaster recovery preparedness, dependency analysis, and recovery validation.
- Develop automation, self-healing capabilities, and AI-assisted operational solutions using Python, Java, AWS services, Amazon Q, Kiro, or related technologies.
- Collaborate with technology and business stakeholders to identify operational risks and prioritize reliability investments.
- Mentor engineers and promote Site Reliability Engineering best practices across the organization.
Benefits
- Competitive compensation, including base pay and annual incentive.
- Comprehensive health and life insurance and well-being benefits, based on location.
- Pension and retirement benefits.
- Paid time off, personal/family care, and other leaves of absence.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
