HPC Systems Administrator

University of Chicago•Austin, TX

1d•Hybrid

About The Position

The University of Chicago Research Computing Center (RCC), a unit in the Office of Research, provides high-end research computing resources to researchers at the University of Chicago. It is dedicated to enabling research by providing access to centrally managed High-Performance Computing (HPC), storage, and visualization resources. These resources include hardware, software, high-level scientific and technical user support, and the education and training required to help researchers make full use of modern HPC technology and local and national supercomputing resources. The Office of Research oversees the conduct of sponsored research, research program development, and contract management functions. The University of Chicago is seeking a highly qualified HPC Systems Security Engineer to join the HPC Systems and Operations team that builds and manages RCC's HPC infrastructure. The individual in this position will be involved in the operation, maintenance, security, and compliance of large-scale complex HPC systems primarily used for research. This position designs automated, scalable, and rapidly deployable solutions to infrastructure development and server configuration. Works independently to install, configure, and maintain operating systems. Uses best practices and systems knowledge to monitor and alert systems, utility software, and firewalls. Guides maintenance for production servers as well as Windows and Linux servers. This is a hybrid position requiring at least 3 days working onsite.

Requirements

Minimum requirements include a college or university degree in related field.
Minimum requirements include knowledge and skills developed through 5-7 years of work experience in a related job discipline.

Nice To Haves

Linux system administration experience in a large, distributed computing environment.
Demonstrated experience and knowledge of system security and best practices.
Knowledge of Linux administration, preferably RHEL/Rocky.
Administration of GPU infrastructure, such as tuning, driver updates, performance monitoring, etc.
Solid skills in scripting with Python or Bash.
Installing, configuring, and managing job schedulers, such as Slurm, Torque, PBS, and LSF.
Automation tools such as Ansible, Puppet, Chef, Salt.
Provisioning tools, including xCAT, Confluent, and Warewulf.
Implementing monitoring tools, such as CheckMK, Zabbix, Nagios, Prometheus, and Grafana.
Working, documenting and enforcing controls required to protect controlled unclassified information, such as NIST 800-53, NIST 800-171, NIST SP 800-223, and FIPS.
Knowledge of at least one distributed storage system, including Storage Scale, Lustre, Gluster, BeeGFS, Ceph, and practical experience.
Demonstrate a working knowledge of InfiniBand concepts.
Writing precise and concise documentation, standard operating procedures.
Understand and translate researchers' scientific goals into computational requirements.
Work well with faculty and researchers.
Identify and gain expertise in appropriate new technologies and/or software tools.
Function as part of an interactive team while demonstrating self-initiative to achieve project's goals and Research Computing Center's mission.
Strong analytical skills and problem-solving ability.

Responsibilities

Designs, deploys, configures, and administers CPU/GPU HPC clusters, including management and compute nodes, storage infrastructure, interconnects such as InfiniBand, and physical infrastructure in the datacenter and related systems.
Monitors, configures, maintains and tunes GPU nodes for optimal performance and utilization following state-of-the-art practices for the required workloads.
Assists with the development and implementation of monitoring and observability tools and infrastructure, collection and aggregation of metrics, development of dashboards.
Develops, maintains, and enforces security procedures and system documentation for operational and compliance purposes.
Tunes, secures, and maintains the HPC job scheduling environment, including fair-sharing, accounting, and policy enforcement.
Troubleshoots and resolves operational, performance, and issues across HPC hardware and software stacks.
Coordinates with hardware and software vendors to address defects, vulnerabilities, and performance issues.
Assists the Computational Scientists team with user support and helpdesk tickets, including elevated support for security-protected environments.
Assists with the implementation and maintenance of secure and reliable backup, archival, disaster-recovery, and restore capabilities for systems and research data.
Performs vulnerability scanning, patch management, system and firmware updates across the infrastructure.
Maintains complex systems and network administration functions.
Works with moderated guidance to administer simple systems and assists in the administration of larger systems.
Maintains all supporting documentation for comprehensive operating system, hardware and software configuration.
Monitors primary responses for information technology related security incidents and violations.
Keeps current with new security and network monitoring technologies, applicable laws and regulations.
Plans and installs necessary patches and upgrades for servers and their associated storage, network, communications, and peripheral sub-systems.
Installs and maintains and appropriate level of intrusion detection, monitoring, and auditing software as required.
Tracks compliance and maintains documentation for hardware, software, and service inventories for management reports.
Performs other related work as needed.