AI Performance Engineer

Cornelis Networks, Inc. • San Jose, CA

2d • Remote

About The Position

Cornelis Networks is seeking an AI Performance Engineer to optimize distributed AI training and multi-node inference across next-generation networking silicon and systems, including adapters, switches, and supporting software. The engineer will collaborate with architecture, firmware, software, and customers to turn laboratory measurements into field-proven performance results. This position is fully remote for employees residing in the United States, with less than 10% travel expected for partner and client meetings, conferences, and company gatherings.

The role focuses on measuring and improving end-to-end performance across CPU and GPU clusters and multiple network fabrics. It includes benchmarking AI workloads and frameworks, designing distributed inference topologies, tuning training and inference, validating networking technologies, profiling and visualizing performance, and communicating findings to customers and internal audiences.

Requirements

• Bachelor’s degree in computer science, electrical engineering, computer engineering, mathematics, or a related field; a master’s degree is preferred.
• Two to three or more years running AI/ML at cluster scale.
• Proven ability to set up, run, and analyze AI benchmarks, with strong understanding of message passing, collectives, scaling efficiency, and performance bottlenecks in training and inference.
• Hands-on experience with distributed training beyond a single GPU, including data, tensor, and pipeline parallelism, ZeRO, FSDP, and sharded optimizers.
• Experience with distributed inference architectures, including replicated versus sharded deployments, tensor or KV parallelism, and MoE.
• Practical experience with AI stacks and communications technologies, including PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, and llama.cpp.
• Comfort with compilers such as GCC, LLVM, Intel, or OneAPI, and MPI stacks.
• Proficiency with Python and shell.
• Familiarity with network architectures such as Omni-Path/OPA, InfiniBand, and Ethernet/RDMA/RoCE.
• Linux systems performance-tuning experience, including NIC offloads, CQ moderation, pacing, and ECN.
• Excellent written and verbal communication skills, including the ability to turn measurements into persuasive customer communications.

Nice To Haves

• Cluster scheduler experience, such as SLURM or PBS.
• Hands-on profiling and tracing of GPU and communication paths using tools such as Nsight Systems, Nsight Compute, ROCm tools, rocprof, roctracer, VTune, or perf.
• Background in HPC performance engineering or storage, including BeeGFS, Lustre, or NVMe-oF.

Responsibilities

• Measure and optimize end-to-end performance for distributed AI training and multi-node inference across CPU and GPU clusters and diverse fabrics, including Omni-Path, Ethernet, and InfiniBand.
• Benchmark, characterize, and tune open-source and industry workloads and benchmarks on current and future compute, storage, and network hardware.
• Design and optimize distributed serving topologies, including sharded and replicated deployments, tensor and pipeline parallelism, MoE expert placement, continuous and adaptive batching, KV-cache sharding or offload, and prefix caching.
• Optimize training and inference, including validating RDMA and GPUDirect RDMA, congestion control, and collective versus point-to-point tradeoffs during inference.
• Develop clear performance comparisons between Cornelis Omni-Path and competing interconnects, and translate results into narratives for sales, marketing, and customers.
• Instrument and visualize performance using profiling tools, application timers, and database visualization tools such as Prometheus and Grafana.
• Share best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level