ML Research Engineer, Performance Optimization
About The Position
Lila Sciences is hiring an ML Research Engineer for its AI Research team to maximize iteration speed for mid- and post-training runs on 1T+ parameter large language models using the latest NVIDIA chips. The role focuses on maximizing GPU utilization for individual training runs and making performance optimizations maintainable, reusable, and adaptable so researchers can confidently use the latest optimized stack. The work spans GPU compute and cluster performance, large-scale data handling, and reinforcement-learning pipelines. The engineer will establish tested, accessible, adaptable performance configurations for the research team and collaborate with researchers to assess throughput and accuracy tradeoffs and determine optimal operating points.
Requirements
- Bachelor's or master's degree in computer science or a related quantitative field and at least 4 years of software engineering or machine learning engineering experience, or a PhD in a relevant research area and at least 1 year of work experience.
- Experience with LLM inference libraries such as vLLM and training libraries such as Megatron, including their use and throughput optimization.
- Experience running LLM inference and/or training at scale across multi-node GPU clusters.
- Familiarity with core reinforcement learning concepts.
- Proven software engineering excellence, including clean APIs, well-tested code, and thoughtful abstractions.
- Hands-on experience supporting and maintaining production libraries used by other engineers and researchers.
- Strong CI/CD skills, including designing integration test suites, build pipelines, and automated quality gates for multi-package ecosystems.
- Passion for agentic systems and awareness of current approaches in LLM tool use and autonomy.
Nice To Haves
- Experience with kernel DSLs and frameworks such as CUDA, CuTe, CuTile, or Triton.
- Experience with GPU networking frameworks such as NCCL and NIXL, and fabrics such as NVLink, IB, and EFA.
- Experience with distributed systems or compute frameworks such as Ray, Flyte, or Airflow.
- Familiarity with containerized compute backends such as Docker and Modal, and sandbox orchestration.
- Experience with Ray actors, Ray Serve, or similar actor-model frameworks.
Responsibilities
- Profile GPU compute performance and develop kernels, handle long contexts, and optimize memory footprints.
- Optimize GPU cluster performance, including collective and MoE frameworks, communication and compute overlap, RDMA/IB/EFA usage, and P/D disaggregation.
- Manage mid-training-scale data across storage, streaming, tokenization, and processing.
- Optimize the asynchronous reinforcement-learning cycle, including environment and sandbox interfacing, long rollouts, large token metadata such as router replay, and weight synchronization.
- Ensure optimized configurations are tested, accessible, and adaptable to research tasks so the team can work efficiently.
- Communicate with researchers about throughput and accuracy tradeoffs, such as lower-precision compute and higher asynchronous staleness, to establish optimal operating points.
Benefits
- Competitive base compensation with bonus potential and generous early-stage equity.
- U.S. full-time employees receive medical, dental, and vision coverage.
- Employer-paid life and disability insurance for U.S. full-time employees.
- Flexible time off and generous company-wide holidays for U.S. full-time employees.
- Paid parental leave for U.S. full-time employees.
- Educational assistance program for U.S. full-time employees.
- Commuter benefits, including bike-share memberships for office-based U.S. employees.
- Company-subsidized lunch program for U.S. full-time employees.
- Regionally tailored benefits for full-time employees outside the U.S.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
