Senior Software Engineer, AI Cluster Networking Performance Engineer
About The Position
The Cisco Validated Infrastructure Services (CVIS) team establishes and proves the performance of large AI clusters before they are handed over to customers. This involves deploying each cluster, validating it against performance baselines, and delivering evidence that it is ready for service. The platform supporting this work is full stack, combining workflow and API services, network and infrastructure automation, and a UI for running validation across labs and customer environments. As a Senior Software Engineer, you will be responsible for debugging, profiling, and optimizing the full AI cluster networking and GPU performance stack. This includes areas from boot-time and PCIe behavior through the Linux kernel, SmartNIC/DPU, RDMA, NCCL, and application-level collective communication.
Requirements
- Bachelors + 7 years of related experience, or Masters + 4 years of related experience, or PHD +1 year of related experience or equivalent related work experience.
- 4+ years of experience working with Linux kernel, networking, GPU, and distributed-systems debugging skills.
- Experience writing code with C/C++/Python.
- Experience with user-mode and kernel-mode drivers.
- Experience with Linux network stack, RDMA/RoCEv2, NCCL, PCIe, and performance profiling.
- Prior experience working with reason from packet captures, traces, counters, logs, and reproducible benchmarks.
Nice To Haves
- SmartNIC/DPU optimization and DOCA SDK experience.
- Cisco switching, AI cluster network fabrics, multi-plane/ToR designs, and high-speed Ethernet or InfiniBand environments.
- Familiarity with NVIDIA diagnostic tooling, Nsight/profiling tools, kernel tracing, and observability platforms.
Responsibilities
- Investigate kernel-level failures, boot-time anomalies, PCIe link issues, driver behavior, and network latency.
- Diagnose and tune RDMA/RoCEv2, NCCL, SmartNIC/DPU, NIC queues, congestion, throughput, and collective communications.
- Read and correlate unified hardware traces across host CPUs, PCIe switches, SmartNICs, and GPU compute streams.
- Capture and analyze Wireshark packets, system logs, NVIDIA support logs, traces, profiles, and observability data.
- Partner with hardware and software teams to eliminate end-to-end bottlenecks.
Benefits
- medical, dental and vision insurance
- a 401(k) plan with a Cisco matching contribution
- paid parental leave
- short and long-term disability coverage
- basic life insurance
- grants of Cisco restricted stock units
- 10 paid holidays per full calendar year
- 1 floating holiday for non-exempt employees
- 1 paid day off for employee’s birthday
- paid year-end holiday shutdown
- 4 paid days off for personal wellness
- 16 days of paid vacation time per full calendar year (non-exempt employees)
- flexible vacation time off program (exempt employees)
- 80 hours of sick time off provided on hire date and each January 1st thereafter
- up to 80 hours of unused sick time carried forward from one calendar year to the next
- Additional paid time away may be requested to deal with critical or emergency issues for family members
- Optional 10 paid days per full calendar year to volunteer
- annual bonuses (for non-sales roles)
- performance-based incentive pay (for sales roles)
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
