Research Scientist - Pre-training

Pluralis Research • San Francisco, CA

12d • Remote

About The Position

Pluralis Research works on Protocol Learning: training and serving large models in a fully decentralized way on small consumer-grade devices connected via the internet. The company has made advances including Agora, a permissionless run that pretrained an 8B model from scratch on consumer GPUs spread over the internet, with no single participant holding the full weights. Protocol Learning introduces challenges such as communication-efficient training across parallelism axes, fault tolerance as nodes join and drop mid-run, heterogeneous compute and networks, and robustness to malicious participants.

As a Research Scientist, you will build the training system that takes Protocol Learning from the 8B run to frontier scale: large models on heterogeneous hardware, in physically different regions, connected by ordinary internet. The role involves distributed pretraining, performance optimization, elasticity and fault tolerance, and instrumentation for monitoring throughput, bottlenecks, and model quality across devices.

Pluralis Research is remote-first, with team members distributed globally and main teams in Australia and North America. Applicants should be comfortable working across time zones and have professional-level English proficiency, written and spoken. The company offers significant ownership for key technical contributors alongside a high base salary, and optional full visa sponsorship and relocation support to Australia or the US.

Requirements

• Hands-on experience training models across many devices in PyTorch with FSDP, DeepSpeed, Megatron, or a self-developed implementation.
• Understanding of data, tensor, and pipeline parallelism.
• Production-quality Python engineering skills.
• Experience with concurrency, failure handling, and profiling before optimizing.
• Evidence of execution through shipped systems, research code, open-source work, or serious personal projects.
• Belief that Protocol Learning is a viable third path for collective, trustless, and sovereign AI.
• Professional-level English proficiency, written and spoken.
• Comfort working across time zones.

Nice To Haves

• Hands-on experience training or serving large language models such as Nemotron, Qwen, or OLMo.
• Experience with peer-to-peer networking and NAT traversal.
• Experience with post-training and reinforcement learning.
• Experience with inference and serving systems.
• Experience at proprietary, open-weight, or open-source AI labs.

Responsibilities

• Implement and optimize model-parallel training, including data, pipeline, and tensor parallelism for large models on heterogeneous GPUs under low-bandwidth, high-latency links.
• Implement techniques that reduce communication overhead while maintaining model convergence in challenging network environments.
• Make training runs resilient to node churn through robust checkpointing, state synchronization, and recovery as participants join and leave.
• Build monitoring that shows throughput, bottlenecks, and model quality across hundreds of devices.

Benefits

• Significant ownership for key technical contributors in addition to a high base salary.
• Flexible remote-first work environment with team members distributed globally.
• Optional full visa sponsorship and relocation support to Australia or the US.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed