Senior Research Engineer

Recruiting From Scratch•San Francisco, CA

2d•Hybrid

About The Position

Our client is a nonprofit AI research institute focused on developing open-source foundational AI models, datasets, and research artifacts. The organization sits at the intersection of academic research, frontier AI labs, and startup-style engineering, giving researchers and engineers the resources to work on large-scale AI systems while maintaining significant freedom to publish, collaborate, and openly share their work. The team develops and releases large open models across language, multimodal, reasoning, and agentic AI, with flagship model families and research projects spanning pretraining, post-training, reinforcement learning, evaluation, datasets, and model infrastructure. The organization has access to substantial compute resources and large-scale GPU infrastructure, allowing Research Engineers to work on problems that require significant distributed training and systems expertise. The team is intentionally relatively small compared with large commercial AI labs, giving engineers substantial ownership and direct collaboration with researchers and technical leadership. The Senior Research Engineer will work closely with leading AI researchers and engineers to build and train foundation models at scale, contributing across the model lifecycle from data and infrastructure through pretraining, post-training, evaluation, and release. The ideal candidate is someone who has actually built and trained foundation models, rather than someone whose experience is primarily using existing models through APIs, RAG systems, or domain-specific fine-tuning.

Requirements

6+ years of general software engineering experience
4+ years of experience in machine learning infrastructure and/or large-scale model training
Strong hands-on experience building and training foundation models
Experience training LLMs and/or multimodal models end-to-end at scale
Experience working on large-scale deep learning systems in production or research environments
Strong software engineering fundamentals alongside deep ML expertise
Experience owning technically complex projects from design through implementation and experimentation
Experience working closely with research scientists and ML engineers
Experience operating in highly technical, research-oriented environments
Experience working with large GPU clusters or distributed training infrastructure
Experience with one or more stages of the foundation-model lifecycle
Strong experience with at least one of: Foundation-model pretraining, Multimodal model training, Post-training, Reinforcement learning, Model architecture, Pretraining data, Agentic model training, Synthetic data, Training infrastructure optimization, CUDA/compute optimization
Must be based in or willing to relocate to Seattle
Must be comfortable working on-site in Seattle
Expert-level Python
Strong PyTorch experience
Experience training large-scale deep learning models
Experience with modern LLM architectures
Experience with transformer-based models
Understanding of mixture-of-experts architectures
Experience with large-scale model pretraining and/or post-training
Experience with distributed training
Experience with GPU computing
Experience with ML infrastructure
Experience with large-scale data processing
Experience building training pipelines
Experience with model evaluation
Strong software engineering and testing practices
Ability to work across the full ML stack
Strong understanding of the full foundation-model lifecycle
Ability to translate research ideas into working software
Ability to design experiments and evaluate results
Ability to identify high-impact research and engineering problems
Strong research intuition combined with practical engineering ability
Ability to move between research and production-quality engineering
Comfortable working with ambiguous research questions
Ability to build systems that support rapid experimentation
Strong understanding of model behavior and evaluation
Ability to reason about tradeoffs between model quality, compute, data, and engineering complexity
Comfortable working closely with research scientists
Ability to communicate research insights clearly
Strong interest in open AI research
Ability to contribute to model releases, datasets, APIs, and technical reports
Comfortable working across infrastructure, training, data, evaluation, and modeling
Ability to take ownership of entire technical projects
Strong interest in advancing foundation-model capabilities
Comfortable learning and applying new research techniques quickly
Extremely high ownership and accountability
Strong technical judgment
Highly proactive and self-directed
Comfortable operating with ambiguity
Excellent written and verbal communication
Strong research and problem-solving ability
Strong analytical thinking
Low-ego and collaborative working style
Comfortable participating in technical and research debates
Able to express strong technical opinions while remaining open-minded
Comfortable receiving and incorporating feedback
Strong attention to detail
Comfortable making decisions with incomplete information
Strong bias toward experimentation and execution
Comfortable working closely with research scientists
Comfortable working across different parts of the ML stack
Strong intellectual curiosity
Fast learner
Comfortable working in rapidly evolving AI research areas
Comfortable owning projects independently
Strong ability to explain complex technical concepts clearly
Comfortable presenting research findings to technical audiences
Resilient and adaptable
Motivated by difficult technical problems
Comfortable working in a small, highly technical team
Strong collaborative instincts
Comfortable contributing outside a narrowly defined specialty
Strong interest in open-source AI
Motivated by advancing the field rather than simply shipping closed commercial products

Nice To Haves

Experience at a leading AI lab, research organization, or technically strong AI company is strongly preferred
Experience at organizations such as Google, Meta, OpenAI, Anthropic, DeepMind, NVIDIA, or comparable AI research/engineering environments is a strong signal
PhD in ML, CS, or a related quantitative field is preferred but not required with equivalent deep learning research experience
Experience contributing to open-source AI research or publications is strongly preferred
Experience with CUDA preferred
Experience with reinforcement learning preferred
Experience with RLVR, GRPO, PPO, OPD, or related methods preferred
Experience building RL environments preferred
Experience with agentic training preferred
Experience with multimodal or vision-language models preferred
Experience with synthetic data generation preferred
Experience with long-context or long-sequence models preferred
Experience with supervised fine-tuning datasets
Experience with instruction tuning
Experience with model distillation
Experience with cloud infrastructure such as GCP or AWS
Experience with Docker
Experience with distributed compute infrastructure
Experience deploying or serving LLMs
Experience with vLLM or SGLang preferred
JAX experience is a plus
Experience with modern RL training frameworks such as SkyRL or Slime is a plus
Strong understanding of model training performance and compute efficiency

Responsibilities

Build, train, evaluate, and improve large language and multimodal foundation models
Work on end-to-end model development across pretraining and post-training
Design and implement large-scale training pipelines
Develop infrastructure for distributed model training and experimentation
Work with modern foundation-model architectures including transformer-based models and mixture-of-experts systems
Develop and optimize training systems running across large GPU clusters
Investigate model behavior through experiments, evaluations, and systematic analysis
Translate research ideas into working implementations and large-scale experiments
Collaborate closely with research scientists and engineers on model development
Own research-engineering projects from initial scoping through implementation, experimentation, and release
Identify high-impact technical and research problems and develop practical solutions
Contribute to improvements across the full model-development lifecycle
Work on model architectures, training methods, datasets, infrastructure, and evaluation
Build systems that enable researchers to run large-scale experiments efficiently
Contribute to pretraining language and multimodal foundation models
Build and improve pretraining datasets and data-processing pipelines
Work on supervised fine-tuning and instruction-tuning datasets
Develop post-training pipelines for reasoning and instruction-following models
Work with reinforcement learning approaches including RL, RLVR, GRPO, PPO, and related methods
Build environments and infrastructure for reinforcement learning
Develop training environments and sandboxing systems for agentic models
Work on synthetic data generation and data-quality pipelines
Contribute to distillation and other model-improvement techniques
Design experiments to improve model capabilities and behavior
Develop evaluation frameworks for measuring model performance
Investigate training failures and model-quality issues
Work across different stages of the model lifecycle depending on project needs
Contribute to emerging approaches for reasoning and agentic model training
Work on multimodal models combining vision and language
Develop and train vision-language models
Contribute to multimodal pretraining and post-training
Build systems for tool use, planning, and long-horizon agentic tasks
Develop reinforcement-learning environments for agents
Build infrastructure for agentic training and evaluation
Work on models capable of reasoning through complex tasks
Investigate model behavior across multimodal and agentic settings
Develop datasets and evaluation systems for multimodal and agentic capabilities
Work with emerging model architectures and training approaches
Collaborate with researchers exploring new capabilities in reasoning, agents, and multimodal AI
Build and optimize infrastructure supporting large-scale model training
Develop distributed training systems for large GPU clusters
Optimize training workloads for performance, reliability, and compute efficiency
Work with CUDA and GPU infrastructure where applicable
Improve training throughput and resource utilization
Build large-scale data-processing and preprocessing pipelines
Develop systems for dataset curation and preparation
Work with cloud infrastructure including GCP and/or AWS
Build and operate containerized ML infrastructure using Docker
Work with distributed computing environments
Deploy and serve large language and multimodal models
Work with model-serving technologies such as vLLM and SGLang
Diagnose infrastructure and training bottlenecks
Improve the reliability and scalability of research infrastructure
Contribute to engineering standards and tooling used across research teams
Bridge research prototypes into production-quality systems
Develop reusable infrastructure that enables researchers to iterate faster
Scope and lead research-engineering projects
Prioritize experiments based on expected research and technical impact
Take ownership of projects from concept through implementation and release
Collaborate closely with research scientists, engineers, and technical leadership
Communicate technical findings and research insights clearly
Participate in technical discussions, design reviews, and research planning
Contribute to open-source models, datasets, libraries, and technical artifacts
Support public model releases and research publications
Develop technical reports and documentation
Share findings with the broader research and engineering community
Contribute to a culture of open research and transparent collaboration
Help shape the direction of future foundation-model research
Work flexibly across different parts of the ML stack based on team needs

Benefits

Competitive benefits
Open to visa transfers including OPT and H-1B transfers
Open to new visa sponsorships including H-1B and TN
E-Verify participant
Welcomes applicants from outside the United States
Opportunity to work with substantial GPU and compute resources
Opportunity to train frontier-scale open foundation models
Direct collaboration with leading AI researchers and engineers
Significant technical and research ownership
Opportunity to publish and openly share research
Opportunity to contribute to open-source AI models and datasets
Opportunity to work across pretraining, post-training, RL, evaluation, and infrastructure
Opportunity to work on language, multimodal, reasoning, and agentic models
Opportunity to work closely with researchers from leading universities and AI organizations
Strong learning environment with access to technical talks and AI experts
Collaborative and transparent research culture
Healthy work/life balance
Generous paid vacation and sick leave
Family leave