Machine Learning Engineer - ML Training Platform

Pluralis Research

3d • Remote

About The Position

Pluralis Research works on Protocol Learning: training and serving large models in a fully decentralized way on small consumer-grade devices connected via the internet. Despite being dismissed as infeasible, we have made significant advances on this problem, most recently Agora, a permissionless run that pretrained an 8B model from scratch on consumer GPUs spread over the internet, with no single participant ever holding the full weights (tech report). While many of the core research problems have been solved, Protocol Learning unlocks a series of new challenges. For the mission in full, read A Third Path: Protocol Learning. Our training and inference doesn't happen in a datacenter. It happens on consumer nodes and cloud instances that are not co-located, connected by ordinary internet, joining and leaving mid-run. Your primary role is to architect, build, and scale the platform that keeps continuous experimentation and large-scale training running on top of that: infrastructure orchestration, distributed compute, and the services that tie them together.

Requirements

• Production experience with infrastructure-as-code (Pulumi/Terraform/CloudFormation) managing multi-cloud deployments, Docker/Kubernetes (EKS), GPU workloads, and heterogeneous clusters at scale.
• Understanding of distributed training workflows: checkpointing, data sharding, model versioning, long-running job orchestration.
• Experience with P2P, NAT traversal, traffic shaping, real bandwidth constraints.
• Strong Python engineering (asyncio, concurrency, retry logic, cloud SDKs, CLI tooling) with hands-on observability and SRE practice; Prometheus/Grafana, performance profiling, incident response.
• Experience in a startup with heavy service orchestration, or at big-tech scale, and ability to show which systems were owned.
• Belief that Protocol Learning is the viable third path for collective, trustless, and sovereign AI.
• Professional-level English proficiency (written and spoken).

Nice To Haves

• Experience with foundation model pre-training, post-training, or RL.
• Experience at proprietary, open-weight and open-source AI labs.

Responsibilities

• Design the resource management systems that provision and orchestrate compute across AWS, GCP, and Azure with infrastructure-as-code (Pulumi/Terraform).
• Handle dynamic scaling, state synchronization, and concurrent operations across hundreds of heterogeneous nodes.
• Architect fault-tolerant infrastructure for distributed ML.
• Build the systems that simulate and handle real network conditions such as bandwidth shaping, latency injection, packet loss.
• Managing node churn and keeping data flowing across workers with heterogeneous connectivity.

Benefits

• Equity-Heavy Package: significant ownership for key technical contributors in addition to a high base salary.
• Remote-First Culture: Flexible work environment with team members distributed globally.
• Visa Sponsorship: Optional full visa sponsorship and relocation support to either Australia or the US.
• Open Problems: Training and serving frontier models on hardware you don't control, over networks you don't own, mostly has no published answers yet. You'll write some of the first ones.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed