AI Performance Engineer

Cornelis Networks, Inc. • San Jose, CA

4d • Remote

About The Position

Cornelis Networks is seeking an AI Performance Engineer to optimize training and multi-node inference across next-generation networking silicon and systems, including adapters, switches, and the associated software stack. The engineer will work with architecture, firmware, software, and customers to turn lab measurements into field-proven performance improvements. The role supports fully remote work for employees residing in the United States, with occasional travel for partner and client meetings, conferences, and in-person collaboration; travel is expected to be less than 10%.

Requirements

• Bachelor’s degree in computer science, electrical engineering, computer engineering, mathematics, or a related field; a master’s degree is preferred.
• 2–3+ years running AI/ML at cluster scale.
• Ability to set up, run, and analyze AI benchmarks, with strong understanding of message passing, collectives, scaling efficiency, and bottleneck analysis for training and inference.
• Hands-on experience with distributed training beyond a single GPU, including data, tensor, and pipeline parallelism, ZeRO, FSDP, or sharded optimizers.
• Hands-on experience with distributed inference architectures, including replicated versus sharded deployments, tensor or KV parallelism, and MoE.
• Practical experience with PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, and llama.cpp.
• Comfort with compilers such as GCC, LLVM, Intel, or OneAPI; MPI stacks; Python; and shell scripting.
• Familiarity with Omni-Path/OPA, InfiniBand, Ethernet/RDMA/RoCE, and Linux performance tuning, including NIC offloads, CQ moderation, pacing, and ECN.
• Excellent written and verbal communication skills, including the ability to turn measurements into customer persuasion.

Nice To Haves

• Experience with cluster schedulers such as SLURM or PBS.
• Hands-on profiling and tracing of GPU and communication paths using tools such as Nsight Systems, Nsight Compute, ROCm tools/rocprof/roctracer, VTune, or perf.
• Background in HPC performance engineering or storage technologies such as BeeGFS, Lustre, or NVMe-oF.

Responsibilities

• Measure and optimize end-to-end performance for distributed AI training and multi-node inference across CPU and GPU clusters and fabrics including Omni-Path, Ethernet, and InfiniBand.
• Benchmark, characterize, and tune open-source and industry workloads and benchmarks on current and future compute, storage, and network hardware, using frameworks such as Megatron-LM, TorchTitan, Composer, vLLM, and llama.cpp.
• Design and optimize distributed serving topologies, continuous and adaptive batching, KV-cache sharding or offload, and prefix caching.
• Optimize training and inference, including validating RDMA/GPUDirect RDMA, congestion control, and collective versus point-to-point tradeoffs during inference.
• Develop comparisons of Cornelis Omni-Path and competing interconnects, and translate results into narratives for sales, marketing, and customers.
• Instrument and visualize performance using profiling tools, application timers, and Prometheus/Grafana.
• Share best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level