Research Scientist

True Talent Solutions Partners•San Francisco, CA

16h•$150,000 - $200,000•Onsite

About The Position

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. The work helps AI labs and enterprise teams understand model capabilities and make informed decisions before deploying models in production. The Research Scientist evaluates newly released AI models, builds benchmarks from the ground up, develops scalable methods for automatically evaluating generated text, and works with engineering teams to implement evaluation methods in production systems. The role also involves collaborating with AI labs and enterprise customers to translate evaluation needs into research initiatives.

Requirements

At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation methods, NLP, or language models; applied experience is valued.
A record of published research, ideally related to NLP or benchmarking.
Strong Python skills and experience building clean, maintainable software.
Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques.
Experience designing benchmarks and translating research findings into practical, scalable systems.
A computer science, machine learning, or related academic background; a bachelor's degree with relevant applied experience may be considered.
Experience collaborating on engineering teams, including Git workflows, code review, and incorporating feedback.

Nice To Haves

Familiarity with Django, Flask, or similar Python HTTP frameworks.
A master's or PhD is preferred.

Responsibilities

Evaluate newly released AI models across a range of capabilities and use cases.
Build benchmarks from the ground up, including dataset development, labeler coordination, and research publication.
Develop scalable and accurate methods for automatically evaluating generated text.
Work with engineering teams to implement evaluation methods in production systems.
Collaborate with AI labs and enterprise customers to translate evaluation needs into research initiatives.