Scientific Evals

Edison Scientific • San Francisco, CA

3d • $130,000 • Onsite

About The Position

About Edison Scientific focuses on building and commercializing AI agents for science, and shares FutureHouse’s mission to build an AI Scientist- scaling autonomous research, productizing it, and applying it to critical challenges such as drug development. Role We are seeking an ambitious, scientifically grounded person to join our team focused on developing rigorous benchmarks and training datasets that advance AI capabilities in biology. This role sits at the intersection of biology, data curation, and machine learning, and is ideal for someone with deep scientific training who is excited to shape how frontier AI systems learn to do science.

Requirements

• Have graduate-level training in biology, biochemistry, computational biology, or a related field, with hands-on research experience.
• Have working knowledge of machine learning concepts, particularly deep learning and large language models.
• Are comfortable with Python and can build workflows for data processing, analysis, and experimentation.
• Possess strong scientific taste and can identify what distinguishes expert-level reasoning from surface-level pattern matching.
• Are detail-oriented and willing to take on high-value but occasionally tedious work.
• Are energized by ambiguous, open-ended problems that require creativity, collaboration, and first-principles thinking to solve.
• Are organized and communicative, able to manage multiple workstreams and coordinate across teams.

Nice To Haves

• Prior experience creating evaluation datasets, annotation guidelines, or working on human-in-the-loop data pipelines.
• Experience with bioinformatics pipelines, biological databases, or sequence analysis tools.
• Hands-on experience fine-tuning or evaluating large language models, or familiarity with RLHF and preference-based training.
• Publications or research experience in areas relevant to AI for science.

Responsibilities

• Design benchmarks that capture the complexity of real biological research, drawing on your domain expertise to identify what makes scientific reasoning hard. This will include open-ended scientific benchmarks and building on prior work like LAB-Bench and BixBench.
• Curate and vet biological datasets to ensure scientific rigor.
• Analyze model outputs, identify failure modes, and contribute to iterative improvements in both datasets and evaluation criteria.
• Collaborate with AI/ML researchers to translate scientific intuition into training signal, helping AI systems learn not just facts but how scientists think.
• Coordinate operations and manage workflows, including working with domain experts, tracking task progress, and maintaining documentation.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level