Senior Research Engineer, LLM Training & Post-Training

Lightning AI•San Francisco, NY

20h•Hybrid

About The Position

Lightning AI is seeking an experienced Senior Research Engineer to join their Research Engineering team. This role will focus on advancing the training, fine-tuning, evaluation, and deployment of large language models (LLMs) on the Lightning AI platform and for customer workloads. The position involves working across model training, post-training, PyTorch, distributed systems, and AI systems engineering to enhance model quality, training efficiency, and developer productivity. The ideal candidate will have deep experience with transformer-based language models, strong software engineering skills, and a passion for improving models and the systems that power them. This role is hybrid, requiring a minimum of two days per week in the San Francisco, Seattle, NYC, or London offices, with fully remote options for candidates outside these hubs. Occasional team and company offsites are expected.

Requirements

Significant experience training, fine-tuning, evaluating, and optimizing transformer-based language models using PyTorch.
Experience with modern LLM training and post-training techniques such as continued pretraining, SFT, RLHF, preference optimization (DPO, PPO, GRPO), reward modeling, or similar approaches.
Strong understanding of distributed training and multi-GPU systems, with experience improving training performance, scalability, or efficiency.
Strong software engineering fundamentals, including building production-quality Python software and research tooling.
Experience designing experiments, evaluating model performance, and debugging complex training or optimization issues.
Excellent communication and collaboration skills, including the ability to work effectively across research, product, infrastructure, and customer-facing engagements.
Comfortable working in fast-moving, ambiguous environments where priorities evolve over time.
Master's degree, PhD, or equivalent industry experience in Machine Learning, AI, Computer Science, or a related field.

Nice To Haves

Experience with one or more of the following: DeepSpeed, FSDP, Megatron-LM, Hugging Face Transformers, TRL, PEFT, Lightning Fabric, or similar training frameworks.
Experience with CUDA, Triton, vLLM, SGLang, TensorRT, or other AI systems and performance optimization technologies.
Experience with GPU performance optimization, mixed precision, memory optimization, or distributed training optimization.
Open-source contributions, research publications, or production AI platforms supporting large-scale training or inference workloads.
Startup experience or experience working on highly cross-functional engineering teams.

Responsibilities

Design, build, and optimize training and post-training pipelines for large language models.
Improve model quality through supervised fine-tuning, continued pretraining, preference optimization, reinforcement learning, evaluation, and experimentation.
Build and improve PyTorch-based training infrastructure, tooling, and developer workflows.
Optimize distributed training across multi-GPU environments by improving throughput, memory efficiency, scalability, and GPU utilization.
Investigate challenging model training issues, including convergence, instability, communication overhead, and performance bottlenecks.
Design evaluation methodologies, benchmark models, analyze failure modes, and guide model improvements through experimentation.
Collaborate directly with customers to understand real-world workloads and translate those learnings into improvements across Lightning AI's research platform.
Partner closely with research, infrastructure, and platform engineering teams to build production-ready AI systems.
Contribute to open-source projects through new features, tooling improvements, documentation, and community engagement.

Benefits

Comprehensive Health Coverage: Medical, dental, and vision coverage for employees and eligible dependents.
Meaningful Equity: RSUs that give employees a stake in the company's long-term success.
Retirement Savings: 401(k) matching (U.S.) and pension contributions (U.K.).
Flexible Time Off: Unlimited PTO, company holidays, and floating holidays to support work-life balance.
Company-Wide Winter Break: Two weeks of company closure each winter to disconnect and recharge.
Paid Parental & Family Leave: Paid leave to support you and your family through life's important moments.
Professional Development: Annual learning and development allowance to support your professional growth.
Wellness Benefits: Wellness and work-from-home stipends to support your physical and mental well-being.
Sabbatical Program: Four weeks of paid sabbatical leave after four years of service.
Flexible Work: Flexible schedules and a hybrid work model for our office-based teams.
In-Office Meals: Complimentary meals at our office hubs.