Cloud & Infrastructure - AIOps Engineer (AI Infrastructure Engineer)

Vailexa • Santa Clara, CA

2d

About The Position

This role works across Linux clusters, SLURM services, GPU infrastructure, and operational tooling that support large-scale AI and high-performance computing. The engineer helps identify system issues before they slow validation and supports the smooth operation of large-scale Linux, SLURM, and GPU systems. The role calls for strong systems troubleshooting, sound operational judgment, and a practical approach to improving system health during deployment and validation.

Requirements

• 4+ years of experience in site reliability engineering, infrastructure engineering, systems engineering, or production operations in large-scale Linux environments.
• Strong Linux administration and troubleshooting skills across services, processes, networking, storage, resource contention, and system performance.
• Experience operating or troubleshooting bare-metal, HPC, or GPU clusters, including SLURM administration and an understanding of cluster health and workload execution.
• Practical Python or Bash scripting skills and familiarity with Ansible or similar tools for operational tasks.
• Experience with monitoring, telemetry, logs, dashboards, alerts, or diagnostic tooling such as Prometheus, Grafana, Alertmanager, or similar platforms.
• Demonstrated troubleshooting and remediation experience, including investigating failures, identifying root causes, and restoring healthy system behavior.
• A bachelor’s degree in computer science, computer engineering, electrical engineering, or a related field, or equivalent practical experience.

Nice To Haves

• Experience with GPU telemetry and SLURM metrics.
• Experience with bare-metal cluster management platforms.
• Practical knowledge of bare-metal provisioning, InfiniBand, RoCE, NVLink or NVSwitch, and parallel storage failure modes.
• Experience supporting early-life cluster operations, system validation, or customer deployment environments.

Responsibilities

• Check system health before validation by verifying node readiness, scheduler health, GPU availability, and critical cluster services.
• Support stable operation during validation by troubleshooting Linux, SLURM, GPU, network, and storage issues, and partnering with appropriate teams to restore healthy system behavior.
• Distinguish infrastructure faults from workload or test issues and turn findings into clear next steps, remediation actions, and operational guidance.
• Improve recurring issues through better diagnostics, runbooks, alerts, and targeted automation that reduces manual effort and helps teams respond more efficiently.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level