Principal Systems Administrator
About The Position
ASRC Federal Data Solutions supports NASA’s High Performance Computing Services program at Ames Research Center in Mountain View, California. The program provides HPC services throughout the lifecycle, including computational requirements, architecture, acquisition, and operations, and supports four on-site supercomputers, more than 18,000 nodes, and systems delivering more than 17 combined petaflops. The team also supports the Human-Systems Integration Division. The position supports infrastructure systems used in HPC operations and is part of the HPC Computer Systems and Storage group, reporting directly to that group’s Manager. The role involves Linux administration and HPC operations, including user support to help resolve issues that affect efficient job execution. It requires deep knowledge of HDR/EDR InfiniBand network fabrics and participation in installing, maintaining, upgrading, and operating HPC systems. Work includes patching, operating-system upgrades, deploying systems, scripting, troubleshooting, testing, benchmarking, and analyzing support tickets for recurring system or user-education issues. The position also requires working with users to identify symptoms and reproduce problems to isolate their causes. The administrator maintains large HPC clusters and production storage, administers and optimizes InfiniBand fabrics, integrates Linux systems with network and HPC software, automates routine work, documents work clearly, and may assist with proposals and publications. After-hours and weekend support may be required.
Requirements
- Bachelor’s degree in computer science or a related field.
- At least 7 years of Linux systems administration experience.
- Strong computer science background and in-depth systems-level knowledge of operating systems and networking.
- Strong ability to analyze and troubleshoot network issues, with understanding of HDR/EDR link speeds, HCAs, switch ports, LIDs, GUIDs, GIDs, subnet management, and fabric topology.
- Strong ability to analyze, debug, and maintain the integrity of an existing code base.
- Demonstrated equivalence of 7 or more years of Linux/UNIX user-support experience and hands-on Linux systems administration experience.
- Superior scripting skills, excellent attention to detail, and proficiency in at least one of Python, Perl, or Bash.
- Strong ability to interact with customers to understand needs, elicit requirements, and obtain feedback on prototype solutions.
- Excellent communication and people skills, and excellent time-management and organizational skills.
- Experience with system configuration management tools such as Ansible, Puppet, or Chef.
- Experience with revision control software such as CVS, SVN, or Git.
- Proficiency in technical writing.
Nice To Haves
- Problem-solving and analysis proficiency for debugging and optimizing applications.
- Familiarity with or proficiency in OpenMP and Message Passing Interface (MPI) programming.
- Experience with Lustre.
- Experience with cloud technologies such as AWS, Azure, or GCP, or with OpenStack or Kubernetes.
Responsibilities
- Maintain HPC clusters with more than 8,000 nodes and production storage exceeding 100 petabytes.
- Administer and operate EDR/HDR InfiniBand fabrics, including switches, Host Channel Adapters (HCAs), cables, transceivers, and related components.
- Monitor fabric health, link status, port errors, link speed and width, congestion, and switch performance.
- Manage Subnet Manager services, including OpenSM or vendor-provided fabric management solutions.
- Maintain fabric topology, routing, addressing, partitioning, and configuration consistency.
- Identify and resolve failed links, degraded ports, disconnected nodes, and fabric initialization issues.
- Establish performance baselines and diagnose packet drops, symbol errors, link recovery events, congestion, and performance degradation.
- Conduct bandwidth and latency testing and optimize fabric routing, topology, and configuration for MPI applications, parallel filesystems, and distributed HPC workloads.
- Install, configure, and maintain InfiniBand drivers, firmware, OFED or inbox RDMA software stacks, and associated libraries.
- Validate HCA compatibility with operating systems, kernels, switch firmware, and application software.
- Troubleshoot RDMA, verbs, MPI, and connectivity issues between compute nodes and storage platforms.
- Support integration with job schedulers, parallel filesystems, and HPC cluster provisioning tools.
- Automate health checks, configuration validation, and routine maintenance using Bash or Python and, where appropriate, Ansible.
- Operate Linux HPC clusters and storage systems, monitor systems proactively, and analyze and correct system issues.
- Develop scripts to automate repetitive tasks or tools that enhance HPC system support.
- Analyze and tune system performance; build, install, and support user-requested software.
- Support evaluation and assessment of new HPC technology.
- Resolve user-reported issues and manage support-ticket requests in Remedy, including reproducing user issues to isolate causes.
- Assist in developing and writing proposals and publications.
- Create and provide clear documentation.
- Provide after-hours and weekend support as required.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
