HPC Infrastructure DevOps Engineer II

St. Jude Children's Research Hospital•Memphis, TN

1d•Hybrid

About The Position

St. Jude Children’s Research Hospital is a world-class pediatric research hospital dedicated to advancing cures and means of prevention for pediatric catastrophic diseases through research and treatment. It is recognized as a 'Best Place to Work' due to its collaborative and positive environment where professionals are supported to advance their careers and contribute to finding cures. The hospital leads the world in understanding, treating, and defeating childhood cancer and other life-threatening diseases. St. Jude is seeking an HPC Infrastructure DevOps Engineer II to join the High-Performance Computing Support (HPCS) team. This role is crucial for the smooth operation, automation, and continuous improvement of St. Jude’s high-performance computing environment, with a strong emphasis on HPC operations, DevOps practices, and automation for configuration, testing, monitoring, and autonomous remediation. The position involves supporting a modern research computing ecosystem that spans both on-premises and remote-site infrastructure. This ecosystem includes HPC compute platforms for research and data-intensive workloads, GPU-enabled environments for AI and machine learning applications, high-capacity research, compliant, and scratch storage tiers, as well as archival, backup, and disaster recovery services. The role also involves operational tooling for observability, governance, and process automation. The HPC Infrastructure DevOps Engineer II will collaborate closely with infrastructure, storage, security, and research teams to deliver reliable and scalable services for computational science, regulated workflows, and AI-enabled research. This position is central to the HPCS service portfolio, encompassing daily HPC client request fulfillment, performance and utilization monitoring, data management and governance, data cataloguing and archival services, and HPC process automation DevOps.

Requirements

Bachelor's degree in Computer Science, Engineering, business or related field of study.
Two (2) years of IT experience with experience in infrastructure operations and engineering environments.
Some experience in infrastructure design, systems analysis, and security management.
Some experience working with business stakeholders to identify and document requirements.
Proven performance in earlier role/comparable role.

Nice To Haves

Master's degree.
Experience supporting Linux-based enterprise or research computing environments.
Experience with scripting or automation using Python, Bash, or similar languages.
Familiarity with DevOps and infrastructure-as-code practices using tools such as Ansible, Terraform, Git-based workflows, or CI/CD platforms.
Experience with observability platforms for logs, metrics, dashboards, and alerting.
Familiarity with HPC workload schedulers such as IBM LSF, Slurm, or comparable systems.
Experience supporting high-performance storage, backup, and archival services.
Familiarity with containers and reproducible compute environments such as Singularity, Docker, or related platforms.
Understanding of secure multi-user platform operations in research or regulated environments.
Experience supporting GPU-based systems for AI and machine learning workloads.
Familiarity with CUDA-enabled environments, GPU monitoring, and NVIDIA software stack dependencies (CUDA, cuDNN, NCCL).
Experience with AI/ML ecosystems such as PyTorch, TensorFlow, Jupyter, and distributed training workflows.
Familiarity with distributed and multi-GPU training frameworks such as PyTorch Distributed, DeepSpeed, Horovod, or Ray.
Understanding of data pipeline, storage throughput, checkpointing, and large dataset staging requirements for model training and inference.
Familiarity with operational practices adjacent to MLOps, including experiment tracking, artifact handling, workflow automation, and workload observability.
Understanding of secure and compliant support for AI workloads operating on sensitive research data.
Ability to troubleshoot AI application issues across infrastructure, scheduler, storage, container, and accelerator layers.
Familiarity with generative AI tools that improve productivity in infrastructure operations, DevOps, and technical support workflows.
Ability to use generative AI assistants to accelerate scripting, automation, troubleshooting, and documentation tasks.
Familiarity with prompt design and iterative prompting techniques for script development, log analysis, workflow generation, and systems diagnostics.
Understanding of the limitations, risks, and verification requirements of generative AI outputs, including accuracy validation, security awareness, and protection of sensitive or regulated data.
Ability to identify practical use cases for generative AI that reduce manual effort across HPC support, governance tracking, and process automation.
Strong troubleshooting and analytical skills.
Service-oriented mindset with excellent communication and follow-through.
Ability to work effectively across infrastructure, operations, and user-facing support functions.
Commitment to documentation, process discipline, and continuous improvement.
Comfort operating in a mission-driven research and regulated data environment.
Ability to balance technical depth with responsiveness in a fast-paced, high-stakes setting.

Responsibilities

Support the day-to-day operation of St. Jude’s HPC infrastructure across compute and storage platforms.
Maintain a stable, secure, and scalable environment for research computing and data-intensive scientific workflows.
Work with downstream operational teams to ensure systems are configured, validated, monitored, patched, and maintained effectively.
Participate in infrastructure testing, upgrade activities, service transitions, and operational readiness efforts.
Contribute to the reliability and supportability of hybrid HPC environments spanning primary and remote-site services.
Respond to daily user requests involving HPC access, Linux environment support, storage allocation, software availability, job troubleshooting, and data movement.
Provide timely and effective support to researchers, analysts, and technical staff using HPC and AI-enabled research resources.
Resolve service incidents and user issues through structured troubleshooting and escalation as needed.
Maintain service-oriented communication with users and stakeholders to support a high-quality support experience.
Implement and improve monitoring for compute nodes, GPU resources, scheduler activity, storage systems, backup operations, and platform health.
Track usage trends, availability, capacity consumption, and operational KPIs to support efficient service delivery.
Analyze utilization patterns and recommend improvements to throughput, performance tuning, scheduling efficiency, and user experience.
Build and maintain dashboards, metrics collection workflows, health checks, and alerting mechanisms to support proactive operations and continuous process improvement.
Support governance reporting and visibility into service consumption and infrastructure health.
Support operational controls for research and compliant data across active storage, protected environments, backup systems, and archival tiers.
Implement and maintain standards for data handling, retention, access control, traceability, and lifecycle operations.
Contribute to governance tracking and reporting for HPC-supported data services.
Assist with data movement and retention workflows across high-performance, compliant, backup, and archival storage platforms.
Support data intake, metadata-aware cataloguing, archival placement, recall, restore validation, and tier-to-tier data movement.
Assist with workflows involving archival platforms, cold storage, backup systems, and long-term retention services.
Improve discoverability and lifecycle management of research datasets through automation and procedural standardization.
Support operational validation of archival and recovery workflows for critical data services.
Use automation tooling to handle system configuration, provisioning, platform maintenance, testing, and operational workflows.
Enable DevOps lifecycle functions by supporting tooling and processes for development, testing, release, and operational support.
Build and maintain CI/CD pipelines and repeatable infrastructure workflows to improve reliability, consistency, and deployment speed.
Reduce manual effort by developing scripts, integrations, and self-service mechanisms for recurring HPCS operational tasks.
Apply automation and generative AI tools responsibly to improve scripting, documentation, incident analysis, and support efficiency.
Assist with deployment and support of AI software stacks, containerized research environments, and Python-based computational workflows.
Help optimize GPU utilization, data throughput, storage access patterns, and job execution for AI training and inference use cases.
Support reproducible environments for AI applications through dependency management and platform validation.
Maintain GPU software stacks, containerized runtimes, dependency consistency, and high-throughput data access for distributed AI training and inference.
Contribute to operational support for research teams using container runtimes, distributed job workflows, and accelerator-aware scheduling.
Identify and deploy security measures through vulnerability assessment, configuration review, patch coordination, and risk-aware operational practices.
Participate in incident response, troubleshooting, and root cause analysis for infrastructure and service disruptions.
Support backup validation, restore readiness, and disaster recovery operational practices for critical HPC services.
Follow institutional requirements for secure handling of research and regulated data.
Work with end users and partner teams to understand operational needs, user requirements, and service KPIs.
Coordinate across technical teams to improve service quality, communication, and execution.
Contribute to ongoing process improvement initiatives that reduce lead time, strengthen platform reliability, and improve user experience.
Maintain accurate technical documentation for systems, configurations, procedures, and knowledge articles.
Perform other duties as assigned to support the goals and objectives of the department and institution.