HPC Linux System Administrator
About The Position
MRI Technologies is hiring an HPC Linux System Administrator to support NASA's Flight Sciences Laboratory at Johnson Space Center under the JETS II contract. The laboratory supports analysis for NASA human spaceflight programs and operates more than 700 machines, 26,000 cores, and 10 petabytes of storage used by more than 1,000 engineers and scientists. The administrator will help operate and improve this on-premises HPC cluster, including its job scheduler and parallel filesystem; help users package and run software environments using HPC containers; and build and support CI/CD workflows on cluster run nodes, including workflows using Jacamar. The role is hands-on and focused on on-premises HPC rather than public-cloud, DevOps-only, or Kubernetes-only work. The position is based onsite at NASA Johnson Space Center in Houston, Texas. Fully remote work may be considered for candidates with strong, demonstrated HPC administration experience. Applicants must be able to provide proof of U.S. citizenship or U.S. permanent residency and complete a U.S. government background investigation. The role has been posted at multiple levels, and candidates may be considered at any advertised level depending on experience and business needs.
Requirements
- Production experience administering an HPC job scheduler such as Slurm, PBS/Torque, or LSF; experience limited to submitting jobs is insufficient.
- Production experience administering a high-speed parallel filesystem such as Lustre or GPFS/Spectrum Scale; general NAS, SAN, or NFS administration is not a substitute.
- At least 5 years of Linux system administration in on-premises, bare-metal, or cluster/research-computing environments, including at least 2 years of HPC administration.
- A bachelor's degree or equivalent certification in a related field, typically with a minimum of 5 years of experience.
- Experience using containers in an HPC setting to package and run user environments on a shared cluster.
- Experience building and supporting CI/CD workflows.
- Experience with system configuration management.
- Experience with system monitoring and alerting.
- Strong problem-solving, planning, and communication skills.
- Ability to work well in a collaborative team environment.
- Ability to provide proof of U.S. citizenship or U.S. permanent residency and complete a U.S. government background investigation.
Nice To Haves
- Experience with Red Hat-based Linux distributions such as RHEL, Rocky, or Alma.
- Familiarity with InfiniBand high-speed networking.
- Experience with cluster provisioning tools such as xCAT or Warewulf.
- Experience with Ansible and/or Foreman for configuration management.
- Experience with log consolidation and monitoring tools, and with Git/GitLab, including CI/CD pipelines.
- Familiarity with Jacamar or comparable tooling for running CI/CD jobs against HPC clusters and run nodes.
- Experience applying AI in system administration, such as integrating AI tooling into operational workflows, automation, or user-facing HPC jobs.
- MPI workflow and administration experience, such as Open MPI, MPICH, or Intel MPI.
- Package management and environment modules experience, such as Lmod/Environment Modules, SPACK, or EasyBuild.
- Knowledge of NASA security mechanisms, including security plans, POAMs, ATOs, and Risk Assessments.
Responsibilities
- Administer and improve the HPC cluster at NASA's Flight Sciences Laboratory.
- Administer the HPC job scheduler, including configuration, accounting, node states, and troubleshooting failed or stuck jobs.
- Administer the high-speed parallel filesystem.
- Help users package and run software environments, including older operating system or package versions, with HPC containers on the shared cluster.
- Build and support CI/CD workflows that run on HPC clusters and cluster run nodes, including Jacamar-based workflows.
- Perform Linux system administration in an on-premises HPC environment.
- Support system configuration management, monitoring, and alerting.
