Senior Software Engineer II, Inference
About The Position
CoreWeave is hiring a Senior Software Engineer II for its inference platform. Senior engineers own areas, lead designs, raise engineering standards, and improve latency, throughput, and reliability across multiple services. The role partners with product, orchestration, and hardware teams to evolve a Kubernetes-native inference platform and meet strict P99 service-level objectives at scale. The engineer will lead architecture and design reviews, own service-level indicators and objectives, and drive reliability improvements through incident follow-up. The position also involves implementing and measuring inference optimizations, strengthening capacity and incident readiness, and mentoring engineers. At the IC4 level, the role owns an area spanning multiple services and teams.
Requirements
- IC3: approximately 3–5 years of industry experience; IC4: approximately 5–8 years of industry experience building distributed systems or cloud services.
- Strong coding skills in Python or Go; C++ is a plus.
- Deep familiarity with networked systems and performance.
- Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks such as Prometheus, Grafana, and OpenTelemetry.
- Practical knowledge of inference internals, including batching, caching, mixed precision (BF16/FP8), and streaming token delivery.
- A proven track record of improving P95/P99 tail latency and service reliability through metrics-driven work.
- Bachelor's or Master's degree in Computer Science or a related field, or equivalent practical experience.
Nice To Haves
- Contributions to inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe.
- Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
- Experience leading multi-team initiatives or partnering with customers on mission-critical launches.
Responsibilities
- Lead design reviews and drive architecture within the team, decomposing multi-service work into clear milestones.
- Define and own SLIs and SLOs; ensure post-incident actions are completed and reliability improves release over release.
- Implement advanced inference optimizations, such as micro-batch schedulers, speculative decoding, and KV-cache reuse, and quantify their impact.
- Strengthen incident readiness through capacity planning, autoscaling policies, graceful degradation, and rollback or traffic-shift strategies.
- Mentor IC1 and IC2 engineers, review cross-team designs, and improve coding and testing standards.
- At IC4, own an area spanning multiple services and teams, such as request routing and adaptive scheduling, cost-per-token analytics, or GPU resource isolation.
Benefits
- Discretionary bonus and equity awards.
- Medical, dental, and vision insurance, 100% paid for by CoreWeave.
- Company-paid life insurance.
- Voluntary supplemental life insurance.
- Short- and long-term disability insurance.
- Flexible Spending Account.
- Health Savings Account.
- Tuition reimbursement.
- Ability to participate in the Employee Stock Purchase Program (ESPP).
- Mental wellness benefits through Spring Health.
- Family-forming support provided by Carrot.
- Paid parental leave.
- Flexible, full-service childcare support with Kinside.
- 401(k) with an employer match.
- Flexible PTO.
- Catered lunch each day at office and data center locations.
- Casual work environment.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
