Research Lead - Pre-training Safety

FAR.AI•Berkeley, CA

3d•$290,000 - $450,000•Hybrid

About The Position

FAR.AI is seeking a Research Lead to spearhead their pre-training safety initiatives. This role involves shaping AI models at their source by focusing on capability control, aiming to remove harmful capabilities while preserving beneficial ones. The goal is to prevent misuse of open-weight models in sensitive areas like CBRN and cyber, and to reduce loss-of-control risks. The position will involve scaling methods like Deep Ignorance to large models (>100B parameters with >1T tokens), partnering with red teams for model testing, and analyzing the scalability of these methods to frontier systems. Research directions include improved data filtering, using gradient routing for capability isolation, actively removing harmful capabilities through methods like unlearning, and incorporating synthetic data to shape model representations and behavior. The Research Lead will build and lead a team, set research direction, mentor staff, and remain hands-on with coding and experiments. This role offers high autonomy in an impact-driven environment focused on empirically grounded, scalable ML safety research.

Requirements

Strong existing research track record in AI or another highly technical subject (e.g., CS, math, physics).
Deep experience with language-model pretraining, dataset construction, or controlled training experiments.
Experience building large-scale pipelines for scoring, filtering, deduplicating, and sampling training corpora.
Strong experimental judgment, including safety–capability evaluations, distribution-shift analysis, and statistically rigorous model comparisons.
Ability to build and debug research systems directly, from classifier fine-tuning through distributed training and evaluation.
Either (a) a clear research agenda with a theory of change, or (b) a strong track record and a research space to develop into an agenda.
Experience leading a team, mentoring graduate students, or supporting early-career researchers.
Ability to effectively communicate novel methods and solutions to both technical and non-technical audiences.
Substantive engagement with the machine learning research field (prior research, employment, or sustained independent contribution).

Nice To Haves

Established publication record in AI safety.
Comfortable writing grant proposals and navigating collaborations with other organizations or external research groups.
Missing key leadership experience or are earlier in your career, consider the Research Scientist pathway.

Responsibilities

Develop and lead work on pre-training safety.
Shape models' capabilities and internal representations at their source.
Focus on capability control: removing harmful capabilities while preserving benign ones.
Scale methods like Deep Ignorance to large models (>100B parameters with >1T tokens).
Partner with red teams to stress-test resulting models.
Analyze how well methods scale to frontier systems.
Explore research directions including improved data filtering, gradient routing, active removal of harmful capabilities, and adding synthetic data.
Build and lead a team of technical staff.
Set the research direction for the team.
Mentor Members of Technical Staff.
Remain hands-on to write code and run experiments.
Articulate a research agenda with a clear theory of change for mitigating catastrophic risks or increasing AI upside.
Lead novel research projects with potentially unclear markers of progress or success.
Share research findings through written content and presentations to drive adoption.
Mentor and coach junior team members in research skills and ML engineering.
Contribute to the FAR.AI intellectual environment and research culture.
Build a research field around the agenda through grants and events.
Connect research to real-world deployments through independent testing and government advising.