Research Scientist, Open Benchmarks Red Team

Snorkel AI•New York City, NY

About The Position

Snorkel AI is hiring a researcher to support its Open Benchmarks Red Team program, which works to improve the reliability of frontier AI benchmarks. This hands-on research role combines AI evaluation, adversarial testing, benchmark repairs, and collaboration with benchmark authors. The researcher will identify benchmark weaknesses, help develop and validate fixes, and create reusable methods to improve benchmark quality across tasks and datasets. They will share findings with benchmark authors, collaborate on repairs, and support adoption of improved benchmark versions. Snorkel AI’s Open Benchmarks Grants is a $30 million commitment to support researchers and independent builders developing AI evaluations. The program supports work across areas including physical AI, scientific discovery, cybersecurity, safety, and alignment, and helps teams test and strengthen benchmarks before and after public release. The role contributes to technical reports, open-source tools, publications, and new approaches to benchmark evaluation and improvement. Snorkel AI describes an environment where employees can shape priorities, influence strategic decisions, deepen technical expertise, explore leadership opportunities, and develop skills across functions.

Requirements

  • Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
  • Strong Python programming skills and experience building, running, and debugging AI agents or evaluation systems.
  • Ability to identify weaknesses in existing evaluation methods, develop new approaches, and rigorously test whether they work.
  • Experience designing rigorous experiments, analyzing model behavior, and investigating unexpected failures.
  • Familiarity with LLM or agent benchmarks, evaluation frameworks, or automated grading systems.
  • Experience independently driving research and engineering projects from problem identification through implementation and validation.
  • Strong communication and collaboration skills.

Nice To Haves

  • Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
  • Experience developing or evaluating benchmarks for coding agents, computer use, professional workflows, or similar areas.
  • Familiarity with RL environments, agent trajectories, verifiers, and evaluation infrastructure.
  • Open-source contributions or publications in AI evaluation, robustness, or agentic systems.
  • PhD in Computer Science, Machine Learning, or a related field, or equivalent research or industry experience.

Responsibilities

  • Identify and reproduce benchmark issues, including reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities.
  • Propose next steps for validated benchmark issues, implement selected fixes with benchmark authors, and verify fixes without breaking valid solutions.
  • Reuse existing tools where possible and develop new checks to detect recurring benchmark weaknesses.
  • Review findings with benchmark authors before publication, collaborate on repairs, and support official improved releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation and improvement.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service