HPC Systems Engineer

University of Pennsylvania•Philadelphia, PA

About The Position

The Penn Advanced Research Computing Center (PARCC) core facility is seeking a highly qualified and motivated High Performance Computing (HPC) Systems Engineer to join the team. PARCC's main cluster (Betty), delivers HPC, data-intensive science and Artificial Intelligence (AI) resources to researchers at the University of Pennsylvania. The HPC Systems Engineer contributes to the strategic planning, design, testing, organization, and implementation of cutting-edge technology projects for the facility, and leads the systems team.

Requirements

Bachelor's degree and 3-5 years of experience as a systems engineer at an academic institution or equivalent combination of education and experience
Expertise in Infiniband networking
Experience configuring jod and resource management applications (Slurm)
Experience deploying HPC portal (OpenOnDemand, Cryo-em, coldfront)
Familiarity with scientific software deployment (spack, easybuild)
Expertise with cluster management software (xCAT, BCM,)
Experience deploying file systems, troubleshooting, and maintenance
A Cover Letter and Resume/CV are required to be considered for this position. Please upload your Cover Letter where it asks you to upload your Resume/CV; multiple documents are allowed.

Responsibilities

Collaborate with senior staff to design, plan, test, and implement advanced hardware solutions for HPC-AI environments.
Deploy and configure physical hardware using HPC deployment tools and orchestration frameworks (e.g., Ansible).
Ensure high availability and minimal downtime of HPC resources to meet the needs of the research community.
Optimize, monitor, and troubleshoot HPC file systems for performance and reliability.
Conduct system benchmarking and develop automated testing to ensure a robust and efficient HPC infrastructure.
Maintain job scheduling systems and enforce storage allocation policies to ensure equitable use of shared resources.
Administer and configure the Slurm scheduler in alignment with institutional research policies.
Participate in planning sessions related to network and security operations; collaborate with the university’s central networking group (ISC).
Apply HPC networking configurations and security protocols to optimize resource utilization and protection.
Maintain a secure, stable, and evolving system/software environment to support dynamic research requirements.
Implement and manage data security controls, including user- and group-based access.
Operate comprehensive monitoring systems for rapid issue detection and long-term performance analysis.
Automate user account lifecycle processes, including creation, maintenance, and removal.
Install and maintain HPC tools to facilitate processes, for example, coldfront.
Manage hardware and software inventory in coordination with vendors.
Establish a collaboration with other groups to provide and keep information about all assets.
Provide technical guidance on new projects involving HPC-AI computing within the institution.
Develop custom tools as needed, and contribute relevant innovations to open-source communities when appropriate.
Evaluate, implement, and test emerging technologies with potential benefits for the HPC-AI research community.
Continuously assess emerging tools and technologies for integration into current and future HPC cluster environments.
Actively mentor and support the training of new and existing staff under the incumbent’s supervision.
Participate in departmental and university-sponsored training programs to enhance knowledge and skills; supervisor-approved commercial training may be substituted where appropriate.

Benefits

Health, Life, and Flexible Spending Accounts: Penn offers comprehensive medical, prescription, behavioral health, dental, vision, and life insurance benefits to protect you and your family’s health and welfare. You can also use flexible spending accounts to pay for eligible health care and dependent care expenses with pre-tax dollars.
Tuition: Take advantage of Penn's exceptional tuition benefits. You, your spouse, and your dependent children can get tuition assistance here at Penn. Your dependent children are also eligible for tuition assistance at other institutions.
Retirement: Penn offers generous retirement plans to help you save for your future. Penn’s Basic, Matching, and Supplemental retirement plans allow you to save for retirement on a pre-tax or Roth basis. Choose from a wide variety of investment options through TIAA and Vanguard.
Time Away from Work: Penn provides you with a substantial amount of time away from work during the course of the year. This allows you to relax, take vacations, attend to personal affairs, recover from illness or injury, spend time with family—whatever your personal needs may be.
Long-Term Care Insurance: In partnership with Genworth Financial, Penn offers faculty and staff (and your eligible family members) long-term care insurance to help you cover some of the costs of long-term care services received at home, in the community or in a nursing facility. If you apply when you’re newly hired, you won’t have to provide proof of good health or be subject to underwriting requirements. Eligible family members must always provide proof of good health and are subject to underwriting.
Wellness and Work-life Resources: Penn is committed to supporting our faculty and staff as they balance the competing demands of work and personal life. That’s why we offer a wide variety of programs and resources to help you care for your health, your family, and your work-life balance.
Professional and Personal Development: Penn provides an array of resources to help you advance yourself personally and professionally.
University Resources: As a member of the Penn community, you have access to a wide range of University resources as well as cultural and recreational activities. Take advantage of the University’s libraries and athletic facilities, or visit our arboretum and art galleries. There’s always something going on at Penn, whether it’s a new exhibit at the Penn Museum, the latest music or theater presentation at the Annenberg Center, or the Penn Relays at Franklin Field to name just a few examples. As a member of the Penn community, you’re right in the middle of the excitement—and you and your family can enjoy many of these activities for free.
Discounts and Special Services: From arts and entertainment to transportation and mortgages, you'll find great deals for University faculty and staff. Not only do Penn arts and cultural centers and museums offer free and discounted admission and memberships to faculty and staff. You can also enjoy substantial savings on other goods and services such as new cars from Ford and General Motors, cellular phone service plans, movie tickets, and admission to theme parks.
Flexible Work Hours: Flexible work options offer creative approaches for completing work while promoting balance between work and personal commitments. These approaches involve use of non-traditional work hours, locations, and/or job structures.
Penn Home Ownership Services: Penn offers a forgivable loan for eligible employees interested in buying a home or currently residing in West Philadelphia, which can be used for closing costs or home improvements.
Adoption Assistance: Penn will reimburse eligible employees on qualified expenses in connection with the legal adoption of an eligible child, such as travel or court fees, for up to two adoptions in your household.