Principal Systems Engineer, Enterprise Backup & Disaster Recovery Solutions

Neurocrine Biosciences • San Diego, CA

5d • $132,700 - $182,000

About The Position

This is a hands-on, Principal Systems Engineering role responsible for leading the design, implementation, operation and continuous improvement of enterprise data protection and disaster recovery (DR) solutions across cloud, on-prem, hybrid, and scientific systems infrastructure. The Principal Systems Engineer scales backup and recovery coverage across enterprise and lab systems, establishes and operates SOPs to regularly test backups and validate recoverability. The individual in this role defines and implements DR solutions for core infrastructure systems and services. This includes building the foundational DR capability from the ground up, planning and executing DR exercises on a recurring cadence, and maintaining backup, recovery, and DR testing artifacts as objective evidence supporting the enterprise resiliency objectives owned by the Cybersecurity team. This role provides technical leadership and oversight to GxP and compliance processes governing backup, recovery, and disaster recovery (DR), ensuring applicable regulatory requirements are identified and met. Serve as the primary technical resource for RTO/RPO governance, recovery testing, and restore evidence across enterprise IT and lab technology infrastructure. Partner closely with Cybersecurity, Infrastructure, Cloud, Network, and Scientific Systems teams to ensure critical systems and data can be reliably restored within business-approved recovery objectives.

Requirements

• Bachelor’s degree in computer science, Information Technology, Systems Engineering, or a related technical field and 6+ years of progressive experience as a Systems Engineer, with 3+ years focused specifically on backup, recovery, and disaster recovery engineering, OR Master's degree in a related technical field and 4+ years of experience as listed above, OR 10+ years of a relevant combination of technical education, certifications, training and/or progressively responsible systems engineering experience
• Demonstrated experience building and operating enterprise backup and DR capability across hybrid (cloud and on-prem) environments is required
• Experience planning and executing recurring DR exercises and maintaining test evidence is required
• Deep, hands-on expertise designing, implementing, and operating enterprise backup and recovery solutions across cloud (AWS/Azure) and on-prem/hybrid infrastructure (e.g., NetApp, Veeam, Commvault, AWS Backup, Azure Backup, or equivalent platforms)
• Demonstrated experience building disaster recovery capability from the ground up, including DR architecture, replication strategies, failover/failback design, and recovery runbooks
• Strong understanding of RTO/RPO concepts and experience defining and governing recovery objectives by service tier
• Experience planning, coordinating, and executing DR exercises/tests on a recurring basis, including tabletop exercises and full failover testing
• Experience developing SOPs and test protocols for backup validation and restore testing and maintaining audit-ready evidence of testing outcomes
• Working knowledge of enterprise systems administration (Linux and Windows), storage systems, virtualization, and database platforms as they relate to backup and recovery design
• Familiarity with lab and scientific infrastructure (e.g., instrument systems, LIMS, lab data stores) sufficient to define backup and DR requirements and serve as the technical point of contact for lab systems alongside enterprise IT infrastructure
• Experience with automation and scripting (Python, Bash, PowerShell, or equivalent) to streamline backup validation, DR failover, and evidence collection
• Familiarity with regulatory and business continuity frameworks relevant to a regulated environment (e.g., GxP, SOX, data integrity requirements, business continuity/DR standards) and their translation into technical controls
• Working knowledge of change management frameworks (ITIL or similar) and ITSM platforms such as ServiceNow for tracking DR and backup remediation activities
• Experience partnering with Cybersecurity/risk teams on enterprise resiliency objectives, audits, and control evidence
• Advanced analytical, problem-solving, and technical decision-making skills, with the ability to evaluate complex issues, identify viable solutions, and recommend an appropriate path forward
• Excellent communication skills with the ability to translate complex technical concepts, risks, metrics, and recommendations for technical, business, and leadership audiences
• Strong cross-functional collaboration and influencing skills, with demonstrated ability to lead initiatives across teams without direct authority.
• Excellent project and program management skills, including the ability to manage multiple priorities, dependencies, and deadlines with a high degree of accuracy and effectiveness
• Ability to evaluate risk, criticality, business impact, and technical dependencies to prioritize work and recommend resiliency investments
• Ability to operate independently with limited supervision, determine appropriate methods and approaches, and serve as a go-to resource within area of expertise
• Ability to mentor and provide technical guidance to less-experienced colleagues

Nice To Haves

• Working Experience with Veeam Backup solution is plus
• Certifications such as AWS Certified Solutions Architect, Veeam VMCE, Certified Business Continuity Professional (CBCP), or equivalent are a plus but not required

Responsibilities

• Lead and scale the enterprise data backup and recovery program across cloud, on-prem, hybrid, and lab infrastructure, increasing coverage of critical systems, servers, instruments, and data stores over time
• Serve as the primary technical owner and go-to resource for backup and recovery coverage across both enterprise IT infrastructure and lab/scientific systems, ensuring lab systems, instruments, and associated data stores are identified, scoped, and included in backup and DR planning alongside enterprise systems in AWS and Azure
• Design and implement backup architecture across compute, storage, and database platforms, ensuring backup strategies align with data classification, retention, and recovery requirements
• Partners with Infrastructure, Cloud, Network, and Scientific Systems teams to identify systems and user/instrument generated data lacking adequate backup coverage and drive remediation to close gaps
• Establish and maintain SOPs to regularly test backups (restore testing, integrity validation, sampling strategies) to ensure ongoing reliability and resiliency of the backup solution
• Define and track KPIs for backup health (e.g., backup success rate, coverage percentage, restore test pass rate, time to restore) and report progress to leadership and governance forums
• Troubleshoot and resolve complex backup and recovery failures, driving root cause analysis and long-term corrective actions
• Define and implement disaster recovery solutions and architecture for business critical (Tier 1) technology infrastructure, systems and services, spanning cloud, on-prem, and hybrid environments
• Lead the design and implementation of foundational DR capability where none currently exists, including DR architecture, tooling, runbooks, and recovery procedures for Tier 1 systems and platforms
• Establish critical service tiers in partnership with Cyber, business and technology owners, and recommend and establish corresponding RTO (Recovery Time Objective) and RPO (Recovery Point Objective) targets for each tier.
• Plan and execute DR exercises on a regular, recurring cadence, coordinating cross functional participation across Infrastructure, Network, Cloud, and application teams
• Identify and remediate gaps uncovered during DR testing (architecture, dependencies, documentation, automation) to continuously improve recovery readiness
• Build automation to streamline DR failover/failback processes and reduce manual effort and recovery time during actual DR events
• Maintain a long-term repository of backup, recovery, and DR testing artifacts (test plans, results, restore evidence, remediation records) as objective evidence supporting enterprise resiliency objectives owned by the Cybersecurity team
• Lead technical execution of GxP and compliance requirements governing backup, recovery, and DR practices, partnering with Cybersecurity on enterprise resiliency policy, risk tolerance, and audit/compliance requirements (e.g., GxP, SOX, business continuity standards)
• Publish recurring recoverability results and metrics (e.g., percentage of Tier 1 services tested, percentage of successful restorations within approved recovery objectives) to leadership and governance reviews
• Support internal and external audits related to data protection, backup, and disaster recovery, ensuring evidence is complete, current, and readily retrievable
• Serve as the enterprise technical escalation point for backup and DR related incidents, driving recovery execution and post incident review
• Mentor engineers on backup, recovery, and DR engineering practices, and contribute to enterprise architecture reviews to ensure resiliency is designed in early rather than retrofitted
• Other duties as assigned

Benefits

• retirement savings plan (with company match)
• paid vacation, holiday and personal days
• paid caregiver/parental and medical leave
• health benefits to include medical, prescription drug, dental and vision coverage

What This Job Offers

• Job Type: Full-time
• Career Level: Principal