About The Position

NVIDIA is seeking a Software Engineer, Collective Communication and Network Performance to join its team in Santa Clara, California. The team develops software tools that help technical users understand complex computing systems and is building software to automatically detect and diagnose collective communication and network performance problems in AI workloads across thousands of GPUs. The role applies knowledge of NCCL and network fabrics to production software that supports large-scale AI systems. The engineer will create detection algorithms using always-on telemetry, build infrastructure that correlates per-collective records with network counters and topology, and produce diagnoses with supporting evidence. The role also defines data needs across NCCL profiler and tuner plugin interfaces, RDMA transport state, and NIC and switch counters, working with owning teams to deliver the data at production-acceptable overhead. The engineer will reproduce communication problems on internal clusters, validate diagnoses, maintain regression tests, and extend coverage to point-to-point and other communication paths based on real incidents.

Requirements

  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
  • At least 8 years of relevant software engineering experience, including hands-on work on a collective communication library, an RDMA transport, or network performance for GPU or HPC workloads.
  • Working knowledge of how NCCL or a comparable library operates, including algorithms, protocols, channels, transports, topology detection, and their effects on latency and bandwidth.
  • Working knowledge of practical RDMA networking, including InfiniBand or RoCE, NIC and switch counters, congestion control, and measurement tools.
  • Demonstrated experience diagnosing a multi-node communication problem from measurements, proving its cause, and turning the diagnosis into software used by others.
  • Proven history of landing changes in code owned by other teams, from defining requirements through delivery.
  • Strong debugging skills across component boundaries, with changes validated through code review, tests, and independent measurements, including when using automated development tools.
  • Proficiency in C++ and Python.
  • Clear communication of evidence, uncertainty, and next steps to engineers and users.

Nice To Haves

  • Contributions to NCCL, UCX, libfabric, NVSHMEM, or a comparable communication library, or to their profiling and tuner plugins.
  • Experience with point-to-point and expert-parallel communication paths such as NIXL, DeepEP, or NVSHMEM, or with fabrics such as EFA.
  • Experience building detection or root-cause analysis that ran in production, with insight into its precision and recall.
  • Experience operating or debugging training or inference jobs above a thousand GPUs, including hang localization.

Responsibilities

  • Develop detection algorithms that identify communication performance problems from always-on telemetry and distinguish among causes such as a late rank, degraded GPU, fabric congestion, silent NIC fault, slower path choice over NVLink, or a hang.
  • Build analysis infrastructure to ingest per-collective records from many ranks, correlate them with network counters and topology, and produce diagnoses with supporting evidence.
  • Define the data required by detectors from NCCL profiler and tuner plugin interfaces, RDMA transport state, and NIC and switch counters; collaborate with owning teams to deliver it at production-acceptable overhead.
  • Reproduce communication problems on internal clusters and verify that automated diagnoses match known causes.
  • Maintain reproduced cases as regression tests that gate every change.
  • Extend coverage beyond NCCL to point-to-point and other communication paths, guided by evidence from real incidents.

Benefits

  • Equity
  • Benefits
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service