AI Infrastructure Technical Systems Engineer

Cisco•Research Triangle Park, NC

6d•$140,300 - $178,100

About The Position

The Cisco IT AI Infrastructure team delivers high-performance compute and accelerated AI infrastructure as a service across Cisco’s global enterprise footprint. The team architects, builds, and operates enterprise-grade supercomputing platforms that support foundational models, distributed AI training, inference engines, and machine learning platforms. The AI Infrastructure Technical Systems Engineer serves as a principal systems architect and technical authority for enterprise AI platforms. The role establishes architectural direction, leads complex infrastructure initiatives, mentors engineers, and connects accelerated hardware capabilities with high-performance software execution. The engineer collaborates with a global multidisciplinary team, including internal AI researchers, Cisco hardware and network architecture teams, and industry partners such as NVIDIA, AMD, and Red Hat.

Requirements

Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline, with at least 5 years of progressive experience in enterprise systems engineering, HPC, AI infrastructure, or SRE.
Proficiency in Python, Go, or Bash, with extensive experience building declarative automation using tools such as Ansible, Terraform, CI/CD, and GitOps.
Advanced expertise designing and administering production Kubernetes, Red Hat OpenShift, and HPC workload managers such as Slurm, Kueue, Volcano, and Ray.
Strong hands-on understanding of AI backend network fabrics, RDMA technologies such as RoCEv2, and high-speed multi-hundred-gigabit network topologies.
In-depth knowledge of enterprise Linux (RHEL/Ubuntu), kernel module lifecycle, memory subsystems, GPU device drivers, and out-of-band management using Redfish or IPMI.

Nice To Haves

Master’s degree in Computer Science, Computer Engineering, or a related technical discipline, or an additional 2+ years of equivalent specialized engineering experience.
At least 3 years of experience managing enterprise server platforms at scale, such as environments with 500+ nodes, including automated hardware provisioning and centralized infrastructure management.
At least 2 years of hands-on experience deploying, troubleshooting, and optimizing heterogeneous GPU software stacks such as NVIDIA CUDA and AMD ROCm, and executing GPU validation suites in production.
At least 3 years of experience designing and implementing enterprise secrets management architectures such as HashiCorp Vault and integrating identity federation protocols (OIDC, SAML, SSO) across production applications.
At least one active, current industry-recognized advanced certification, such as CKA/CKS, RHCA/RHCE, or an advanced networking credential.
Verifiable contributions to open-source infrastructure projects, Kubernetes SIGs, HPC scheduling communities, or telemetry frameworks, such as authored commits, merged pull requests, or published technical documentation.

Responsibilities

Lead architectural design, hardware qualification, capacity planning, and maintenance of large-scale bare-metal NVIDIA and AMD GPU clusters.
Architect resilient, multi-tenant compute and control planes across bare-metal Kubernetes, Red Hat OpenShift, and NVIDIA BCM.
Collaborate with network engineering to design, benchmark, and optimize high-throughput, low-latency backend fabrics, including RDMA, RoCEv2, and 400G/800G networks.
Champion Site Reliability Engineering practices to reduce operational toil, enable autonomous self-healing, and uphold service-level objectives.
Architect infrastructure-as-code and GitOps deployment pipelines using Ansible, Terraform, Python, Go, and modern CI/CD tooling.
Oversee automation for firmware baselines, BIOS/BMC configuration, kernel driver updates, platform upgrades, and certificate rotations.
Optimize distributed workload scheduling, fair-share queueing, and GPU quota governance across Slurm, Kueue, Volcano, and Ray.
Profile low-level systems performance and diagnose device interconnect bottlenecks, bandwidth utilization, memory hierarchies, and compute efficiency.
Establish automated Day-2 diagnostics and recovery procedures for GPU degradation, driver anomalies, and hardware faults.
Architect end-to-end observability standards using OpenTelemetry, Prometheus, and Splunk, including role-specific dashboards and anomaly detectors.
Serve as the senior escalation authority for high-impact incidents, lead root-cause analyses, and align operations with ITIL frameworks.
Author architecture blueprints and standard operating procedures, and mentor and coach mid-level and senior engineers.

Benefits

Medical, dental, and vision insurance
401(k) plan with Cisco matching contribution
Paid parental leave
Short- and long-term disability coverage
Basic life insurance
Potential Cisco restricted stock unit grants, subject to eligibility and vesting conditions
Paid holidays and a floating holiday for non-exempt employees
Employee birthday day off, paid year-end holiday shutdown, and personal wellness days
Paid vacation or flexible vacation time off, depending on employee classification and eligibility
Sick time off
Additional paid time away may be requested for critical or emergency family issues
Optional paid volunteer days
Annual bonuses for eligible non-sales employees