Principal Systems Administrator
About The Position
ASRC Federal Data Solutions (AFDS) supports NASA’s High Performance Computing Services program at Ames Research Center in Mountain View, California. The program provides HPC services across the lifecycle, including computational requirements, architecture, acquisition, and operations, and supports the Human-Systems Integration Division. The posting describes an immediate opening on the Infrastructure team supporting the systems that enable HPC operations. The successful candidate will support the HPC Computer Systems and Storage group and report directly to its Manager. The role involves Linux systems administration, HPC operations, InfiniBand fabric administration, and support for users working to run jobs efficiently. The administrator will participate in installing, maintaining, upgrading, and operating HPC systems, and will troubleshoot issues by working with users to understand symptoms and reproduce problems. Work may also include testing, benchmarking, scripting, and analyzing support tickets to identify recurring system or user-education issues.
Requirements
- Bachelor’s degree in computer science or a related field.
- Strong computer science background with in-depth systems-level knowledge of operating systems and networking.
- At least 7 years of experience administering Linux systems.
- Equivalent demonstrated experience of 7 or more years in Linux/UNIX user support and hands-on Linux systems administration.
- Strong ability to analyze and troubleshoot network issues, including understanding HDR/EDR link speeds, HCAs, switch ports, LIDs, GUIDs, GIDs, subnet management, and fabric topology.
- Strong ability to analyze, debug, and maintain the integrity of an existing code base.
- Superior scripting skills and proficiency in at least one of Python, Perl, or Bash.
- Strong ability to interact with customers to understand needs, elicit requirements, and obtain feedback on prototype solutions.
- Excellent communication and people skills, time management, and organizational skills.
- Experience with system configuration management tools such as Ansible, Puppet, or Chef.
- Experience with revision control software such as CVS, SVN, or Git.
- Proficiency in technical writing.
Nice To Haves
- Proficiency in analysis and problem-solving for debugging and optimizing applications.
- Familiarity or proficiency with OpenMP and Message Passing Interface (MPI) programming.
- Experience with Lustre.
- Experience with cloud technologies (AWS, Azure, or GCP), OpenStack, or Kubernetes.
Responsibilities
- Maintain HPC clusters with more than 8,000 nodes and production storage exceeding 100 petabytes.
- Administer and operate EDR/HDR InfiniBand fabrics, including switches, Host Channel Adapters (HCAs), cables, transceivers, and related components.
- Monitor fabric health, link status, port errors, link speed and width, congestion, and switch performance.
- Manage Subnet Manager services, including OpenSM or vendor-provided fabric management solutions.
- Maintain fabric topology, routing, addressing, partitioning, and configuration consistency; diagnose and resolve failed links, degraded ports, disconnected nodes, and fabric initialization issues.
- Establish performance baselines and diagnose packet drops, symbol errors, link recovery events, congestion, and performance degradation.
- Conduct bandwidth and latency testing and optimize fabric routing, topology, and configuration for MPI applications, parallel filesystems, and distributed HPC workloads.
- Install, configure, and maintain InfiniBand drivers, firmware, OFED or inbox RDMA software stacks, and associated libraries.
- Validate HCA compatibility with operating systems, kernels, switch firmware, and application software.
- Troubleshoot RDMA, verbs, MPI, and connectivity issues between compute nodes and storage platforms.
- Support integration with job schedulers, parallel filesystems, and HPC cluster provisioning tools.
- Automate health checks, configuration validation, and routine maintenance using Bash or Python and, where appropriate, Ansible.
- Participate in day-to-day operations of Linux HPC clusters and storage systems, including system patching, OS upgrades, deployment of new systems, and troubleshooting.
- Proactively monitor systems, analyze and correct system issues, and perform system performance analysis and tuning.
- Develop scripts to automate repetitive tasks or tools to enhance support of HPC systems.
- Build, install, and support user-requested software; support evaluation and assessment of new HPC technology.
- Resolve user-reported issues and manage support ticket requests in Remedy.
- Interact with users to determine symptoms, reproduce issues, and isolate causes; provide support for users of HPC resources.
- Support testing, benchmarking, tool scripting, and analysis of trouble tickets for patterns indicating system or user-education issues.
- Assist in developing and writing proposals and publications, and create and provide clear documentation.
- Provide after-hours and weekend support as required.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
