Senior Software Engineer II, Inference

CoreWeave • Sunnyvale, WA

1d

About The Position

CoreWeave is hiring a Senior Software Engineer II for its inference team. Senior engineers own areas, lead designs, raise engineering standards, and deliver measurable improvements to latency, throughput, and reliability across multiple services. The role partners with product, orchestration, and hardware teams to evolve a Kubernetes-native inference platform and meet strict P99 service-level objectives at scale.

The engineer will lead technical design and architecture, own reliability metrics and follow-through on incident actions, and implement and measure inference optimizations such as micro-batch scheduling, speculative decoding, and KV-cache reuse. The position also strengthens incident readiness through capacity planning, autoscaling, graceful degradation, and rollback or traffic-shifting strategies. At IC4, the role owns an area spanning multiple services and teams, such as request routing and adaptive scheduling, cost-per-token analytics, or GPU resource isolation.

Requirements

• IC3: approximately 3–5 years of industry experience; IC4: approximately 5–8 years of industry experience building distributed systems or cloud services.
• Strong coding skills in Python or Go; C++ is a plus.
• Deep familiarity with networked systems and performance.
• Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks such as Prometheus, Grafana, and OpenTelemetry.
• Practical knowledge of inference internals, including batching, caching, mixed precision (BF16/FP8), and streaming token delivery.
• A proven track record of improving tail latency (P95/P99) and service reliability through metrics-driven work.
• Bachelor's or Master's degree in Computer Science or a related field, or equivalent practical experience.

Nice To Haves

• Contributions to inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe.
• Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
• Experience leading multi-team initiatives or partnering with customers on mission-critical launches.

Responsibilities

• Lead design reviews and drive architecture within the team; decompose multi-service work into clear milestones.
• Define and own SLIs and SLOs, ensure post-incident actions are completed, and improve reliability release over release.
• Implement advanced inference optimizations, such as micro-batch schedulers, speculative decoding, and KV-cache reuse, and quantify their impact.
• Improve incident readiness through capacity planning, autoscaling policies, graceful degradation, and rollback or traffic-shifting strategies.
• Mentor IC1 and IC2 engineers, review cross-team designs, and raise coding and testing standards.
• At IC4, own an area spanning multiple services and teams, such as request routing and adaptive scheduling, cost-per-token analytics, or GPU resource isolation.

Benefits

• Discretionary bonus and equity awards.
• Medical, dental, and vision insurance, 100% paid for by CoreWeave for eligible US-based full-time employees.
• Company-paid life insurance and voluntary supplemental life insurance.
• Short- and long-term disability insurance.
• Flexible Spending Account and Health Savings Account.
• Tuition reimbursement.
• Ability to participate in the Employee Stock Purchase Program (ESPP).
• Mental wellness benefits through Spring Health.
• Family-forming support through Carrot.
• Paid parental leave.
• Flexible, full-service childcare support with Kinside.
• 401(k) with a generous employer match.
• Flexible PTO.
• Catered lunch each day at office and data center locations.
• Casual work environment.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior