HPC Operations Engineer

LLNL•Livermore, CA

20h•Onsite

About The Position

Lawrence Livermore National Laboratory (LLNL) is hiring an HPC Operations Engineer in the Livermore Computing Division within the Computing Directorate. The role develops, deploys, and maintains tools, automation, and processes that support the availability, reliability, and performance of Livermore Computing’s high-performance computing systems. The engineer applies HPC systems and software development knowledge to monitoring, alerting, and diagnostic solutions for computing clusters, storage systems, networks, and data center facilities, working with system administrators, software developers, and facilities engineers. The position may be filled at SES.1 or SES.2, according to the candidate’s knowledge and related experience. SES.2 includes greater technical project leadership, more complex software and automation development, analysis of operational data, escalation responsibilities, staff guidance, and independent prioritization of concurrent work. The position requires full-time on-site presence and the ability to work all shifts, including overnight Owl shift (12 a.m.–8 a.m.), weekends, and holidays. It requires eligibility to obtain and maintain a U.S. Department of Energy Q-level security clearance, which requires U.S. citizenship.

Requirements

Ability to secure and maintain a U.S. DOE Q-level security clearance; U.S. citizenship is required.
Bachelor’s degree in computer science, computer engineering, or a related field, or an equivalent combination of education and related experience.
Fundamental knowledge of and experience with Linux/UNIX systems and enterprise data storage, including installation, configuration, networking, RAID concepts, system logs, and troubleshooting.
Understanding of programming and scripting languages such as C/C++, Rust, Go, Python, or bash/csh/ksh.
Experience with version control and configuration management systems such as Git, Ansible, or CFEngine.
Fundamental knowledge of system monitoring, alerting, incident management practices, and tools.
Verbal, written, and interpersonal communication skills to collaborate with technical teams and end users and explain technical information.
Ability to work all shifts, including Owl shift (12 a.m.–8 a.m.), weekends, and holidays.
At SES.2, comprehensive knowledge of HPC environments and technologies such as Linux clusters, InfiniBand, Slurm, Lustre, and archival storage systems.
At SES.2, proficient experience developing software tools and automation in C/C++, Python, or similar languages, and applying technical principles to resolve moderately complex system, network, and facility issues.
At SES.2, experience leading technical work packages or discrete projects and mentoring less-experienced staff, and ability to work with limited direction in a dynamic environment with competing priorities.
At SES.2, relevant previous experience and demonstrated ability to perform independent technical work.

Nice To Haves

Experience with monitoring and observability tools such as Grafana, Prometheus, Splunk, or Nagios.
Experience with data center utility infrastructure and building management systems, including power, water, and cooling distribution units supporting HPC environments.
Experience with virtual machines, container technologies, and orchestration such as Docker, Podman, or Kubernetes.
Experience working in a Closed Area or other secure computing environment.

Responsibilities

Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for HPC clusters, storage systems, high-speed networks, and data center facility infrastructure.
Create and maintain software tools and processes that streamline alarm response, diagnostics, and routine operational tasks.
Integrate data from building management and facility systems, including power and cooling, with HPC system monitoring.
Share alarm response and monitoring duties with operations staff; analyze, diagnose, and resolve hardware, software, network, and facility problems; determine root causes and implement corrective actions with relevant teams and vendors.
Participate in deploying, testing, and validating new HPC systems and components, and decommissioning retired systems.
Develop and maintain technical documentation, procedures, and runbooks for operational tools and processes.
Respond to customer problems and questions, coordinating with Hotline, system administration, and other LC groups to improve service for end users.
At SES.2, design and develop moderately complex software tools and automations to improve computing and facility operations, reliability, and efficiency.
At SES.2, analyze system, alarm, and incident data to identify trends and improve monitoring coverage, alarm accuracy, and operational processes.
At SES.2, lead discrete technical projects or work packages, coordinating HPC system deployment, validation, and decommissioning with system administration, facilities, and other LC groups.
At SES.2, diagnose and resolve moderately complex to complex system and facility problems and serve as a technical escalation resource.
At SES.2, develop and deliver training, guide and informally mentor less experienced staff, and independently prioritize concurrent operational and project work.
Perform other duties as assigned.