HPC Operations Engineer

LLNL•Livermore, CA

1d•$121,830 - $185,544•Onsite

About The Position

Lawrence Livermore National Laboratory is hiring an HPC Operations Engineer in the Livermore Computing Division within the Computing Directorate. The role develops, deploys, and maintains tools, automation, and processes that support the availability, reliability, and performance of Livermore Computing’s high-performance computing systems. The engineer works on monitoring, alerting, and diagnostic solutions for HPC clusters, storage systems, networks, and data center facilities, collaborating with system administrators, software developers, and facilities engineers in a team-oriented environment. The position is available at either the SES.1 or SES.2 level, based on knowledge and related experience. Higher-level responsibilities include designing moderately complex software and automation, identifying operational trends, leading discrete technical projects, serving as a technical escalation resource, delivering training, mentoring staff, and independently prioritizing concurrent work. The role requires full-time on-site presence and the ability to work all shifts, including overnight Owl shift (12 a.m.–8 a.m.), weekends, and holidays. A DOE Q-level security clearance is required; eligibility requires U.S. citizenship and a federal background investigation.

Requirements

Ability to secure and maintain a U.S. DOE Q-level security clearance; U.S. citizenship is required for clearance eligibility.
Bachelor’s degree in computer science, computer engineering, or a related field, or an equivalent combination of education and related experience.
Fundamental knowledge of and experience with Linux/UNIX systems and enterprise data storage, including installation, configuration, networking, RAID concepts, system logs, and troubleshooting.
Understanding of programming and scripting languages such as C/C++, Rust, Go, Python, or bash/csh/ksh.
Experience with version control and configuration management systems such as Git, Ansible, or CFEngine.
Fundamental knowledge of system monitoring, alerting, and incident management practices and tools.
Verbal, written communication, and interpersonal skills to collaborate with technical teams and end users and to present and explain technical information.
Ability to work all shifts, including overnight Owl shift (12 a.m.–8 a.m.), weekends, and holidays.
At SES.2: Comprehensive knowledge of HPC environments and technologies such as Linux clusters, InfiniBand, Slurm, Lustre, and archival storage systems.
At SES.2: Proficient experience developing software tools and automation in C/C++, Python, or similar languages, and applying technical principles to moderately complex system, network, and facility issues.
At SES.2: Experience leading technical work packages or discrete projects and mentoring less-experienced staff; ability to work with limited direction amid competing priorities.
At SES.2: Relevant previous experience and demonstrated ability to perform independent technical work.

Nice To Haves

Experience with monitoring and observability tools such as Grafana, Prometheus, Splunk, or Nagios.
Experience with data center utility infrastructure and building management systems, including power, water, and cooling distribution units supporting HPC environments.
Experience with virtual machines, container technologies, and orchestration such as Docker, Podman, or Kubernetes.
Experience working in a Closed Area or other secure computing environment.

Responsibilities

Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for HPC clusters, storage systems, high-speed networks, and data center infrastructure.
Create and maintain software tools and processes to streamline alarm response, diagnostics, and routine operational tasks.
Integrate data from building management and facility systems, including power and cooling, with HPC monitoring systems.
Share alarm response and monitoring duties with operations staff; analyze, diagnose, and resolve hardware, software, network, and facility problems, determine root causes, and implement corrective actions with relevant teams and vendors.
Participate in deployment, testing, and validation of new HPC systems and components, and decommission retired systems.
Develop and maintain technical documentation, procedures, and runbooks.
Respond to customer problems and questions, collaborating with Hotline, system administration, and other groups to improve end-user service.
At SES.2: Design and develop moderately complex software tools and automation to improve operations, reliability, and efficiency.
At SES.2: Analyze system, alarm, and incident data; identify trends; and implement strategies to improve monitoring coverage, alarm accuracy, and operational processes.
At SES.2: Lead discrete technical projects or work packages, coordinating with system administration, facilities, and other groups.
At SES.2: Diagnose and resolve moderately complex to complex system and facility problems and serve as a technical escalation resource.
At SES.2: Develop and deliver training, guide and informally mentor less-experienced staff, and independently prioritize concurrent work.
Perform other duties as assigned.