Machine Learning Engineer, AI

Biohub•Redwood City, CA

4h•$214,000 - $301,000•Hybrid

About The Position

Biohub is building a general-purpose system to accelerate scientific discovery by integrating frontier AI models, biological foundation models, and laboratory capabilities. The organization operates a large AI compute cluster dedicated to biology and works across three frontier research institutes. Its research includes molecular modeling, biological and imaging foundation models, reasoning and agentic systems, mechanistic interpretability, and large-scale scientific data systems. As an ML Engineer, you will work with researchers and infrastructure engineers to build systems that enable scientific research. The work includes training and evaluating biological foundation models, building shared libraries and data systems, optimizing GPU workloads, maintaining distributed training infrastructure, supporting model deployment, and improving monitoring and testing. You may also collaborate with partner researchers and engineers to deploy models and infrastructure in external environments.

Requirements

Significant experience in at least one of the listed areas.
Strong Python, PyTorch, and CUDA skills, including custom training loops, and a solid understanding of model architectures, optimization, and evaluation.
Experience with large-scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
Ability to diagnose performance problems across GPUs, memory, networking, storage, and data loading.
Experience building large-scale data pipelines and shared research tools.
Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.

Nice To Haves

Familiarity with scientific data formats such as Zarr or HDF5.

Responsibilities

Work with researchers to train, fine-tune, and evaluate biological foundation models and diagnose training problems.
Help evaluate and curate training and validation datasets at petabyte scale.
Design and create libraries and abstractions for researchers and engineers.
Optimize GPU-native data loading for large tensor workloads and resolve hardware utilization bottlenecks across backends.
Build and maintain reliable distributed pre-training and post-training systems.
Support production deployment and track model and dataset artifacts throughout the ML lifecycle.
Improve model and cluster monitoring, unit and integration testing, and continuous integration.
Work with partner researchers and engineers to deploy models and supporting infrastructure in external environments.