AI Infrastructure Technical Systems Engineer

Cisco • San Jose, NC

6d

About The Position

Cisco IT’s AI Infrastructure team delivers high-performance compute and accelerated AI infrastructure as a service across Cisco’s global enterprise footprint. The team architects, builds, and operates enterprise-grade supercomputing platforms that support foundational models, distributed AI training, inference engines, and machine learning platforms.

The AI Infrastructure Technical Systems Engineer and Technical Leader serves as the principal systems architect and technical authority for enterprise AI platforms. The role sets architectural direction, leads complex infrastructure initiatives, mentors engineering talent, and connects accelerated hardware capabilities with high-performance software execution. It works with a global, multidisciplinary team, including internal AI researchers, Cisco hardware and network architecture teams, and industry partners such as NVIDIA, AMD, and Red Hat.

Requirements

• Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline.
• At least 5 years of progressive experience in enterprise systems engineering, HPC, AI infrastructure, or SRE.
• Proficiency in Python, Go, or Bash, with extensive experience building declarative automation using Ansible, Terraform, CI/CD, and GitOps.
• Advanced expertise designing and administering production Kubernetes, Red Hat OpenShift, and HPC workload managers such as Slurm, Kueue, Volcano, and Ray.
• Strong hands-on understanding of AI backend network fabrics, RDMA technologies such as RoCEv2, and high-speed multi-hundred-gigabit network topologies.
• In-depth knowledge of enterprise Linux (RHEL/Ubuntu), kernel module lifecycle, memory subsystems, GPU device drivers, and out-of-band management (Redfish, IPMI).

Nice To Haves

• Master’s degree in Computer Science, Computer Engineering, or a related technical discipline, or an additional 2+ years of equivalent specialized engineering experience.
• 3+ years managing enterprise server platforms at scale, such as environments with 500+ nodes, including automated hardware provisioning and centralized infrastructure management solutions.
• 2+ years of hands-on experience deploying, troubleshooting, and optimizing heterogeneous GPU software stacks such as NVIDIA CUDA and AMD ROCm, and running GPU validation suites in production.
• 3+ years designing and implementing enterprise secrets management architectures such as HashiCorp Vault and integrating identity federation protocols (OIDC, SAML, SSO) across production applications.
• At least one active, current industry-recognized advanced certification, such as CKA/CKS, RHCA/RHCE, or advanced networking credentials.
• Demonstrable contributions to open-source infrastructure projects, Kubernetes SIGs, HPC scheduling communities, or telemetry frameworks, such as authored commits, merged pull requests, or published technical documentation.

Responsibilities

• Lead architectural design, hardware qualification, capacity planning, and maintenance for large-scale bare-metal NVIDIA and AMD GPU clusters.
• Architect resilient, multi-tenant compute and control planes across bare-metal Kubernetes, Red Hat OpenShift, and NVIDIA BCM HPC cluster managers.
• Collaborate with network engineering to design, benchmark, and optimize high-throughput, low-latency backend fabrics using RDMA, RoCEv2, and 400G/800G networking.
• Champion Site Reliability Engineering to reduce operational toil, enable autonomous self-healing, and uphold stringent Service Level Objectives.
• Architect Infrastructure as Code and GitOps deployment pipelines using Ansible, Terraform, Python, Go, and modern CI/CD tooling.
• Oversee automated workflows for firmware baselines, BIOS/BMC configuration, kernel driver updates, platform upgrades, and certificate rotations.
• Optimize distributed workload scheduling, fair-share queueing, and GPU quota governance across Slurm, Kueue, Volcano, and Ray.
• Profile low-level systems performance and diagnose device interconnect bottlenecks, bandwidth utilization, memory hierarchies, and compute efficiency.
• Establish automated Day-2 diagnostics and recovery procedures for GPU degradation, driver anomalies, and hardware faults.
• Architect end-to-end observability standards using OpenTelemetry, Prometheus, and Splunk, including role-specific dashboards and anomaly detectors.
• Serve as senior escalation authority for high-impact incidents, lead root-cause analyses, and align operations with ITIL frameworks.
• Author architecture blueprints and standard operating procedures, and mentor and coach mid-level and senior engineers.

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• 401(k) plan with Cisco matching contribution
• Paid parental leave
• Short- and long-term disability coverage
• Basic life insurance
• Paid holidays
• Floating holiday for non-exempt employees
• Paid birthday day off
• Paid year-end holiday shutdown
• Personal wellness days
• Vacation time or flexible vacation time off, depending on employee classification
• Sick time off
• Additional paid time away may be requested for critical or emergency family issues
• Optional paid volunteer days
• Annual bonuses for non-sales roles, subject to Cisco policies
• Employees may be eligible for Cisco restricted stock unit grants, subject to continued employment and vesting terms

What This Job Offers

• Job Type: Full-time
• Career Level: Senior