Senior Research Engineer, LLM Training & Post-Training

Lightning AI•Seattle, WA

13h•$165,000 - $310,000•Hybrid

About The Position

This role will focus on advancing how large language models are trained, fine-tuned, evaluated, and deployed across Lightning AI's platform and real-world customer workloads. It will work across model training, post-training, PyTorch, distributed systems, and AI systems engineering to improve model quality, training efficiency, and developer productivity while collaborating closely with researchers, infrastructure engineers, and customers. We're looking for someone who enjoys turning cutting-edge research into production systems. You have deep experience training and improving transformer-based language models, strong software engineering fundamentals, and a passion for solving difficult problems across model training, evaluation, and AI systems. Rather than building applications on top of existing models, you're motivated by improving the models themselves and the systems that power them. Our work spans models that power the Lightning AI platform, customer-specific model workloads, and research that translates into reusable training and platform capabilities.

Requirements

Significant experience training, fine-tuning, evaluating, and optimizing transformer-based language models using PyTorch.
Experience with modern LLM training and post-training techniques such as continued pretraining, SFT, RLHF, preference optimization (DPO, PPO, GRPO), reward modeling, or similar approaches.
Strong understanding of distributed training and multi-GPU systems, with experience improving training performance, scalability, or efficiency.
Strong software engineering fundamentals, including building production-quality Python software and research tooling.
Experience designing experiments, evaluating model performance, and debugging complex training or optimization issues.
Excellent communication and collaboration skills, including the ability to work effectively across research, product, infrastructure, and customer-facing engagements.
Comfortable working in fast-moving, ambiguous environments where priorities evolve over time.
Master's degree, PhD, or equivalent industry experience in Machine Learning, AI, Computer Science, or a related field

Nice To Haves

Experience with one or more of the following: DeepSpeed, FSDP, Megatron-LM, Hugging Face Transformers, TRL, PEFT, Lightning Fabric, or similar training frameworks.
Experience with CUDA, Triton, vLLM, SGLang, TensorRT, or other AI systems and performance optimization technologies.
Experience with GPU performance optimization, mixed precision, memory optimization, or distributed training optimization.
Open-source contributions, research publications, or production AI platforms supporting large-scale training or inference workloads.
Startup experience or experience working on highly cross-functional engineering teams.

Responsibilities

Design, build, and optimize training and post-training pipelines for large language models.
Improve model quality through supervised fine-tuning, continued pretraining, preference optimization, reinforcement learning, evaluation, and experimentation.
Build and improve PyTorch-based training infrastructure, tooling, and developer workflows.
Optimize distributed training across multi-GPU environments by improving throughput, memory efficiency, scalability, and GPU utilization.
Investigate challenging model training issues, including convergence, instability, communication overhead, and performance bottlenecks.
Design evaluation methodologies, benchmark models, analyze failure modes, and guide model improvements through experimentation.
Collaborate directly with customers to understand real-world workloads and translate those learnings into improvements across Lightning AI's research platform.
Partner closely with research, infrastructure, and platform engineering teams to build production-ready AI systems.
Contribute to open-source projects through new features, tooling improvements, documentation, and community engagement

Benefits

Comprehensive Health Coverage: Medical, dental, and vision coverage for employees and eligible dependents.
Meaningful Equity: RSUs that give employees a stake in the company's long-term success.
Retirement Savings: 401(k) matching (U.S.) and pension contributions (U.K.).
Flexible Time Off: Unlimited PTO, company holidays, and floating holidays to support work-life balance.
Company-Wide Winter Break: Two weeks of company closure each winter to disconnect and recharge.
Paid Parental & Family Leave: Paid leave to support you and your family through life's important moments.
Professional Development: Annual learning and development allowance to support your professional growth.
Wellness Benefits: Wellness and work-from-home stipends to support your physical and mental well-being.
Sabbatical Program: Four weeks of paid sabbatical leave after four years of service.
Flexible Work: Flexible schedules and a hybrid work model for our office-based teams.
In-Office Meals: Complimentary meals at our office hubs.