Cloud & Infrastructure - AIOps Engineer (AI Infrastructure Engineer)

Vailexa • Santa Clara, CA

3d

About The Position

The AIOps Engineer works across Linux clusters, SLURM services, GPU infrastructure, and operational tooling that support large-scale AI and high-performance computing. The role focuses on identifying system issues before they slow validation and keeping large-scale systems operating smoothly through strong troubleshooting, sound operational judgment, and practical improvements to system health during deployment and validation.

Requirements

• At least 4 years of experience in site reliability engineering, infrastructure engineering, systems engineering, or production operations in large-scale Linux environments.
• Strong Linux administration and troubleshooting skills across services, processes, networking, storage, resource contention, and system performance.
• Experience operating or troubleshooting bare-metal, HPC, or GPU clusters, including SLURM administration and understanding cluster health and workload execution.
• Practical Python or Bash scripting skills and familiarity with Ansible or similar tools for operational tasks.
• Experience with monitoring, telemetry, logs, dashboards, alerts, or diagnostic tooling such as Prometheus, Grafana, Alertmanager, or similar platforms.
• Demonstrated troubleshooting and remediation experience, including investigating failures, identifying root causes, and restoring healthy system behavior.
• A bachelor’s degree in computer science, computer engineering, electrical engineering, or a related field, or equivalent practical experience.

Nice To Haves

• Experience with GPU telemetry and SLURM metrics.
• Experience with bare-metal cluster management platforms.
• Practical knowledge of bare-metal provisioning, InfiniBand, RoCE, NVLink or NVSwitch, and parallel storage failure modes.
• Experience supporting early-life cluster operations, system validation, or customer deployment environments.

Responsibilities

• Check system health before validation by verifying node readiness, scheduler health, GPU availability, and critical cluster services.
• Support stable operation during validation by troubleshooting Linux, SLURM, GPU, network, and storage issues and partnering with the appropriate teams to restore healthy system behavior.
• Distinguish infrastructure faults from workload or test issues, and turn findings into clear next steps, remediation actions, and operational guidance.
• Improve recurring issues through better diagnostics, runbooks, alerts, and targeted automation that reduces manual effort and helps teams respond more efficiently.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level