Research Systems Administrator

Universities of Wisconsin • Madison, WI

12d • $75,000 • Onsite

About The Position

The cyberinfrastructure team at WIPAC at the University of Wisconsin-Madison is seeking a Linux system administrator to join our computing and operations team. Our team supports the operations of the IceCube Neutrino Observatory, a particle detector located at the geographic South Pole. In addition, we have significant resources that support data analysis (15+ PB of storage, thousands of CPUs and GPUs) distributed around the world which requires sophisticated infrastructure orchestration. The successful applicant will work directly with the computing systems that operate our detector. This comprises a testing and staging system at UW-Madison as well as the operational system at the South Pole. You will work closely with a dedicated team of scientists and software developers to ensure that our computing system maintains the highest possible uptime in one of the harshest locations on the planet. In addition, you will work with computing and storage systems that support delivery of science-ready data to our collaborators around the world. Your focus will be on operating system deployment, configuration management, and monitoring of these systems. In particular, automated operating system deployment and configuration management with Puppet will be a strong focus. You will also have opportunities to work with large scale cluster storage (Ceph) as well as high-throughput and grid computing systems (HTCondor and the Open Science Grid). In addition to the team at WIPAC, you will work with a diverse international community of researchers in a variety of scientific disciplines to ensure that our computing systems meet the current and future needs of our researchers, as well as adapting to incorporate new technology and platforms over time. We have faculty, students, and staff from around the globe, and routinely work with international colleagues. Opportunities for professional development, conferences, and additional training and education are provided. This position will work in our office in Madison, WI. Flexible schedules and partial remote work options are available.

Requirements

• Prior polar experience, as travel to the geographic South Pole is required. One to three years of Linux system administration experience
• Experience with a scripting language such as Bash, Python, or Ruby
• Experience with free and open-source ecosystems, particularly with automated deployment and configuration management tools in those ecosystems.
• Willingness and ability to deploy to the South Pole for 1 to 3 months per year. This requires passing a comprehensive physical examination
• System automation and configuration management experience (Puppet, Ansible, Chef)
• Experience with monitoring systems (Nagios, Check_MK, Prometheus, Grafana)

Nice To Haves

• TCP/IP networking (IPv6 experience is a plus)
• IoT or embedded systems
• Hardware deployment and management

Responsibilities

• Deploy and manage server hardware in both the South Pole and South Pole Test systems.
• Deploy and manage Linux operating systems in the South Pole System and South Pole Test System
• Automate management of these systems with automated configuration management (Puppet)
• Maintain the configuration management
• Deploy and maintain monitoring systems to ensure system health and adequate capacity
• Automation of other parts of our infrastructure
• Provides expertise and experience in CI to lead troubleshooting, analysis, operation, and support of advanced computing systems and for research
• Serves as a CI domain expert, provides direct support of users, other staff providing support of users, and advanced computing systems
• Provides consultation and expertise for research proposals.
• May act as technical lead for cyber infrastructure projects.
• May serve as Co-Principal Investigator for a proposal
• Writes and updates documentation and tutorials supporting CI activities for research.
• May provide brief courses for internal researchers and external collaborators
• Develops, programs, and/or deploys automation workflows for deployment, configuration, and/or monitoring of systems/services
• Develops unique and custom Cyber Infrastructure solutions for research needs

Benefits

• Opportunities for professional development, conferences, and additional training and education are provided.
• Flexible schedules and partial remote work options are available.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees