Senior Software Engineer, AI Cluster Networking Performance Engineer
Cisco•Milpitas, CA
•$167,700 - $245,200
About The Position
Cisco Validated Infrastructure Services (CVIS) establishes and proves the performance of large AI clusters before they are handed over to customers. We deploy each cluster, validate it against performance baselines, and deliver the evidence that it is ready for service. The platform behind that work is full stack. It combines workflow and API services, network and infrastructure automation, and the UI used to run validation across labs and customer environments.
Requirements
- Bachelors + 7 years of related experience, or Masters + 4 years of related experience, or PHD +1 year of related experience or equivalent related work experience.
- 4+ years of experience working with Linux kernel, networking, GPU, and distributed-systems debugging skills.
- Experience writing code with C/C++/Python and experience with user-mode and kernel-mode drivers.
- Experience with Linux network stack, RDMA/RoCEv2, NCCL, PCIe, and performance profiling.
- Prior experience working with reason from packet captures, traces, counters, logs, and reproducible benchmarks.
Nice To Haves
- SmartNIC/DPU optimization and DOCA SDK experience.
- Cisco switching, AI cluster network fabrics, multi-plane/ToR designs, and high-speed Ethernet or InfiniBand environments.
- Familiarity with NVIDIA diagnostic tooling, Nsight/profiling tools, kernel tracing, and observability platforms.
Responsibilities
- Investigate kernel-level failures, boot-time anomalies, PCIe link issues, driver behavior, and network latency.
- Diagnose and tune RDMA/RoCEv2, NCCL, SmartNIC/DPU, NIC queues, congestion, throughput, and collective communications.
- Read and correlate unified hardware traces across host CPUs, PCIe switches, SmartNICs, and GPU compute streams.
- Capture and analyze Wireshark packets, system logs, NVIDIA support logs, traces, profiles, and observability data.
- Partner with hardware and software teams to eliminate end-to-end bottlenecks.
Benefits
- medical, dental and vision insurance
- a 401(k) plan with a Cisco matching contribution
- paid parental leave
- short and long-term disability coverage
- basic life insurance
- 10 paid holidays per full calendar year
- 1 floating holiday for non-exempt employees
- 1 paid day off for employee’s birthday
- paid year-end holiday shutdown
- 4 paid days off for personal wellness
- 16 days of paid vacation time per full calendar year (non-exempt)
- flexible vacation time off program (exempt)
- 80 hours of sick time off provided on hire date and each January 1st thereafter
- optional 10 paid days per full calendar year to volunteer
- annual bonuses (for non-sales roles)
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
