Lead Principal Site Reliability Engineer

Oracle • United States,

9d • $96,300 - $264,100

About The Position

The Lead Principal Site Reliability Engineer serves as a senior technical authority and consultant for the reliability, architecture, and operation of large-scale, distributed, business-critical platforms. The role defines reliability strategy and standards, influences system design, and ensures services are scalable, observable, resilient, supportable, and capable of meeting availability objectives. It also provides technical direction and mentorship across site reliability, software, platform, and operations teams.

The position leads service reliability and observability efforts, including service-level indicators and objectives, error budgets, operational health metrics, monitoring, logging, tracing, alerting, dashboards, and reliability reporting. It develops automation and platform capabilities to reduce operational toil and improve deployment, provisioning, incident response, and service recovery. The role partners with software teams on continuous integration and delivery, contributes production-quality software and reusable platform components, and promotes infrastructure-as-code and repeatable engineering practices.

The engineer provides technical leadership for severe production incidents, coordinates recovery and stakeholder communications, leads blameless post-incident reviews, and drives long-term corrective work. Responsibilities also include capacity planning, performance analysis, demand forecasting, resilience and disaster-recovery planning, and security and operational governance in partnership with security and compliance teams. The position may participate in an on-call rotation or provide senior escalation support as required.

Oracle describes its work as bringing together data, infrastructure, applications, and expertise, with AI embedded across products and services. The company states that it supports an inclusive workforce and offers competitive benefits and volunteer programs.

Requirements

• Extensive professional experience in site reliability engineering, software engineering, cloud infrastructure, platform engineering, systems engineering, or a related technical discipline.
• Demonstrated experience designing, operating, and improving highly available production systems at significant scale.
• Deep knowledge of distributed systems, cloud architecture, networking, operating systems, storage, databases, and service dependencies.
• Advanced experience with at least one major cloud platform, such as Oracle Cloud Infrastructure, Amazon Web Services, Microsoft Azure, or Google Cloud Platform.
• Strong experience with containerization and orchestration technologies, including Docker and Kubernetes.
• Proven expertise with infrastructure-as-code and configuration-management technologies such as Terraform, Ansible, Chef, Puppet, or equivalent tools.
• Experience implementing observability solutions using metrics, logs, traces, dashboards, and automated alerting.
• Strong programming or scripting skills in one or more languages such as Python, Go, Java, JavaScript, Bash, or similar.
• Experience with continuous integration, continuous delivery, automated testing, and modern release-management practices.
• Demonstrated leadership during critical production incidents and complex technical investigations.
• Ability to diagnose difficult system issues across applications, infrastructure, networks, databases, and cloud services.
• Strong written and verbal communication skills, including explaining technical risk and recommendations to engineering leaders and business stakeholders.
• Proven ability to lead cross-functional technical initiatives without relying solely on formal authority.

Nice To Haves

• Experience supporting enterprise-scale cloud services, software-as-a-service platforms, or other high-availability customer-facing systems.
• Experience defining and operating service-level objectives, error budgets, and reliability scorecards.
• Knowledge of chaos engineering, resilience testing, and automated recovery techniques.
• Experience with multi-region, hybrid-cloud, or multi-cloud architectures.
• Familiarity with security frameworks, compliance requirements, and regulated operating environments.
• Experience improving cloud cost efficiency, capacity utilization, or infrastructure performance.
• Contributions to internal engineering standards, technical communities, open-source projects, or industry publications.
• Bachelor’s or advanced degree in computer science, engineering, information systems, or a related field, or equivalent practical experience.

Responsibilities

• Define and drive site reliability engineering strategy for large-scale, distributed, business-critical platforms.
• Establish reliability standards, engineering practices, and operational-readiness requirements across teams.
• Serve as a senior technical authority for reliability, scalability, resilience, performance, and production operations.
• Influence architecture and design to make systems supportable, observable, fault tolerant, and capable of meeting availability objectives.
• Identify systemic reliability risks and lead cross-functional initiatives to address them.
• Provide technical direction and mentorship to reliability, software, platform, and operations engineers.
• Lead technical reviews and promote consistent engineering practices.
• Define and implement service-level indicators, service-level objectives, error budgets, and operational health metrics.
• Develop monitoring, logging, tracing, alerting, and observability strategies; improve alert quality and reduce unnecessary noise.
• Establish dashboards and reporting for service health, performance, capacity, and risk.
• Use production data and reliability trends to prioritize engineering investments and continuous improvement.
• Design and implement automation to reduce manual intervention, operational toil, and human error.
• Build or enhance tools for deployment, configuration management, infrastructure provisioning, incident response, and service recovery.
• Promote infrastructure-as-code, policy-as-code, automated testing, and repeatable deployment practices.
• Partner with development teams to improve continuous integration and continuous delivery pipelines.
• Develop self-healing and automated remediation capabilities where appropriate.
• Contribute production-quality software and reusable platform components using modern programming and scripting languages.
• Provide technical leadership during complex, high-severity production incidents; coordinate diagnosis, containment, recovery, and stakeholder communication.
• Lead blameless post-incident reviews and ensure corrective actions address root causes.
• Identify recurring failure patterns and develop long-term engineering solutions.
• Improve incident-management processes, escalation procedures, runbooks, and recovery playbooks.
• Participate in an on-call rotation or provide senior escalation support for critical services as required.
• Lead capacity planning, performance analysis, load testing, and demand forecasting for critical platforms.
• Identify performance bottlenecks and recommend architectural or operational improvements.
• Design and validate high-availability, disaster-recovery, backup, and business-continuity capabilities.
• Lead resilience testing, failure-mode analysis, game days, and controlled fault-injection exercises.
• Ensure recovery-time and recovery-point objectives are defined, tested, and achievable.
• Partner with security and compliance teams to embed security into infrastructure, automation, and operational practices.
• Support vulnerability remediation, access-control improvements, audit readiness, and secure configuration management.
• Ensure production environments meet standards for change management, data protection, and operational governance.
• Balance reliability, security, delivery speed, cost, and business priorities when recommending technical solutions.

Benefits

• Medical, dental, and vision insurance, including expert medical opinion.
• Short-term and long-term disability coverage.
• Life insurance and AD&D; supplemental life insurance for employees, spouses, and children.
• Health care and dependent care Flexible Spending Accounts.
• Pre-tax commuter and parking benefits.
• 401(k) Savings and Investment Plan with company match.
• Flexible or accrued vacation, depending on employee eligibility and position; accrual terms are specified in the posting.
• 11 paid holidays.
• Paid sick leave: 72 hours upon hire, refreshing annually, with unused balances carrying over up to a 112-hour cap.
• Paid parental leave.
• Adoption assistance.
• Employee Stock Purchase Plan.
• Financial planning and group legal benefits.
• Voluntary benefits, including auto, homeowner, and pet insurance.
• The role may be eligible for bonus, equity, and compensation deferral.
• Volunteer programs that encourage employees to give back to their communities.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed