Machine Learning Engineer, AI
About The Position
Biohub is building a general-purpose system to accelerate scientific discovery by integrating frontier AI models, biological foundation models, and laboratory capabilities. Its work combines AI, large-scale computing, and experimental research to develop tools that accelerate scientific research and ultimately help cure disease. Biohub operates a large AI compute cluster dedicated to biology across three frontier research institutes. The Machine Learning Engineer will work with researchers and infrastructure engineers to build systems that connect the components of this scientific computing effort. The role includes supporting biological foundation model training and evaluation, large-scale data workflows, GPU performance, distributed systems, production deployment, and infrastructure used by researchers. The posting describes future opportunities at Senior and Staff levels; applying does not guarantee contact about a current opening.
Requirements
- Significant experience in at least one of the listed areas.
- Five or more years building ML systems or distributed infrastructure at scale.
- Strong Python, PyTorch, and CUDA skills, including experience with custom training loops and a solid understanding of model architectures, optimization, and evaluation.
- Experience with large-scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
- Ability to diagnose performance problems across GPUs, memory, networking, storage, and data loading.
- Experience building large-scale data pipelines and shared research tools.
Nice To Haves
- Familiarity with scientific data formats such as Zarr or HDF5.
- Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
- Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.
Responsibilities
- Work with researchers to train, fine-tune, and evaluate biological foundation models and diagnose training problems.
- Help evaluate and curate training and validation datasets at petabyte scale.
- Design and create libraries and abstractions for researchers and engineers.
- Optimize GPU-native data loading for large tensor workloads and resolve hardware utilization bottlenecks across backends.
- Build and maintain reliable distributed pre-training and post-training systems.
- Support production deployment and track model and dataset artifacts throughout the ML lifecycle.
- Improve model and cluster monitoring, unit and integration testing, and continuous integration.
- Work with partner researchers and engineers to deploy models and supporting infrastructure in external environments.
Benefits
- Generous employer match on employee 401(k) contributions.
- Paid time off to volunteer at an organization of the employee’s choice.
- Funding for select family-forming benefits.
- Relocation support for employees who need assistance moving.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
