Scientist Jobs

About The Position

Requirements

• BS and a minimum of 3 years relevant industry experience
• Strong experience in evaluating supervised, unsupervised, and deep learning models.
• Hands-on experience evaluating LLMs and using them as scoring/judging mechanisms.
• Familiarity with multimodal models (e.g., image + text, video + audio) and related evaluation challenges.
• Proficiency in Python and libraries such as NumPy, pandas, scikit-learn, PyTorch, or TensorFlow.
• Solid understanding of statistical testing, sampling, confidence intervals, and metrics (e.g., precision/recall, BLEU, ROUGE, FID, etc.).
• Strong documentation skills, including the ability to write technical reports and present to non-technical audiences.

Nice To Haves

• Experience working with open-source evaluation tools like OpenEval, ELO-based ranking, or LLM-as-a-Judge frameworks.
• Familiarity with prompt engineering, few-shot or zero-shot evaluation techniques.
• Experience evaluating generative models (e.g., text generation, image generation).
• Prior contributions to ML benchmarks or public evaluations.
• Strong interpersonal skills.

Responsibilities

• Develop robust methodologies to assess the performance of foundation models (e.g., LLMs, vision-language models, etc.) across diverse tasks.
• Leverage LLMs as judges to perform subjective and open-ended model evaluations (e.g., for summarization, reasoning, or multimodal generation tasks).
• Build, curate, and lead evaluation datasets and benchmarks.
• Collaborate with research, engineering, and product teams to define evaluation goals aligned with user experience and product quality.
• Conduct failure analysis and uncover edge cases to improve model robustness.
• Contribute to our tools and infrastructure to automate and scale evaluation processes.