Lead Principal Core Infrastructure Engineer

Oracle•Nashville, TN

3d•$146,300 - $306,400

About The Position

The Lead Principal Core Infrastructure Engineer mentors teams and leads the architecture of highly scalable, interdependent distributed systems. The role identifies and removes performance and scalability bottlenecks for hyper-scale workloads, defines scalability requirements with stakeholders, designs elastic systems, and advances innovation in data plane platforms. It engineers and oversees fault-tolerant, in-service-upgradable designs; optimizes resilience mechanisms; and establishes SLO-aligned durability and availability standards across dependent services. The position establishes KPIs and advanced telemetry, applies formal verification to complex features, and develops replication and synchronization strategies. It advises on and leads resolution of complex production issues, sets operational readiness and standard operating procedure standards, and directs incident response and root-cause analyses. It also architects security controls, leads remediation and compliance efforts, and delivers enterprise-level automation and change strategies for safe patching, updates, and rollbacks.

Responsibilities

Mentor teams in architecting and designing highly scalable, interdependent distributed systems, including horizontal and vertical scaling, performance, and distributed state management.
Identify performance and scalability bottlenecks and recommend code or system changes for large-scale data processing and high-throughput workloads.
Collaborate with stakeholders to define system scalability requirements that meet customer expectations.
Design elastic, high-impact systems and drive innovation in data plane platforms.
Assess systems against nonfunctional scalability requirements and anticipate future business needs.
Design and oversee fault-tolerant systems that withstand in-service updates using redundancy, replication, and automatic failover.
Design systems to handle service disruptions, including network partitions, while making appropriate consistency, availability, or partition-tolerance choices.
Optimize mechanisms for network unreliability, including load-shedding, throttling, and rate-limiting.
Establish standards for availability and durability and ensure systems adhere to service level objectives.
Define KPIs and telemetry to identify risks, gaps, and cyclical dependencies in interdependent systems.
Drive development and customization of dashboards, telemetry systems, and alerting mechanisms to monitor system health and reliability.
Apply industry correctness-verification standards and formally verify complex features, including through techniques such as TLA+.
Develop data replication and synchronization strategies that protect integrity and availability.
Advise on diagnosing, debugging, and resolving complex issues in active systems.
Develop strategies to prevent interruptions and eliminate the need for customer maintenance windows when resolving issues.
Maintain expertise in system dependencies, dependents, and owned systems to support troubleshooting and performance.
Set operational readiness and standard operating procedure standards, and hold third-party partners accountable for meeting them.
Oversee operational support rotations, guide incident response, and lead root-cause investigations.
Architect security measures for data and applications in multi-tenant environments and lead protection initiatives.
Guide remediation plans for security vulnerabilities.
Ensure cloud infrastructure compliance with industry standards and regulations, and guide project documentation.
Develop enterprise-level automation tools and strategies, such as Infrastructure as Code, and oversee implementation.
Align change-management plans and organizational initiatives for patching, updates, and rollbacks, and design systems to automate these processes.
Lead advanced problem-solving as an escalation point, guide data-driven approaches to ambiguous or novel issues, identify root causes, and implement preventive solutions.
Develop and implement improvements to organizational processes, protocols, and workflows; evaluate their effectiveness and create metrics to measure success.