Member of Technical Staff - Research Scientist

Recruiting From Scratch•San Francisco, CA

1d•Onsite

About The Position

This is an early-stage AI company building technology that helps enterprises evaluate, validate, and deploy increasingly capable AI systems with confidence. The company is focused on solving difficult problems around AI evaluation, benchmarking, and the reliability of language models. Its work helps determine how advanced AI systems perform across real-world enterprise use cases and provides the methodologies needed to measure and improve their quality. The engineering and research teams work at the intersection of applied AI research and production software engineering. You will evaluate new models as they are released, build benchmarks from scratch, develop evaluation methodologies, and work closely with engineering teams to turn research ideas into scalable systems. As a Member of Technical Staff — Research Scientist, you will have significant ownership over how AI systems are evaluated. You will work with large datasets, language models, evaluation pipelines, and automated assessment methods while collaborating with AI labs and enterprise customers. This is a highly hands-on applied research role for someone who enjoys building practical evaluation systems, experimenting with modern AI models, and turning research ideas into production-ready methodologies. The role prioritizes applied impact over purely academic research or publishing papers for its own sake.

Requirements

0–3 years of professional experience in applied AI/ML, research engineering, or a closely related field.
Hands-on experience working with AI/ML systems, language models, benchmarking, or evaluation.
Experience building practical AI/ML systems rather than exclusively conducting theoretical research.
Experience working in collaborative engineering or research environments.
Strong interest in evaluating and understanding the behavior of modern AI systems.
Experience designing experiments, methodologies, or measurement frameworks is highly valuable.
Startup, AI lab, research lab, or founding/early-stage company experience is strongly preferred.
Candidates with strong academic backgrounds and limited industry experience can also be considered.
Strong Python programming skills.
Ability to write clean, maintainable, production-quality code.
Experience with AI/ML frameworks such as PyTorch or TensorFlow.
Understanding of machine learning fundamentals.
Familiarity with language modeling and modern generative AI techniques.
Experience working with datasets and data-processing pipelines.
Strong analytical and experimental skills.
Familiarity with benchmarking or evaluation methodologies.
Experience with cloud infrastructure, ideally AWS.
Familiarity with Django, Flask, or other Python-based HTTP frameworks is a plus.
Understanding of software engineering best practices, Git, pull requests, code review, and collaborative development.
Hands-on experience with large language models or other modern AI systems.
Strong interest in LLM evaluation and benchmarking.
Experience evaluating model outputs or AI-generated content.
Understanding of automated evaluation methodologies.
Ability to design experiments for comparing AI models or systems.
Familiarity with benchmark construction and dataset design.
Understanding of model failure modes and evaluation challenges.
Experience with language modeling, NLP, or related AI research is strongly preferred.
Ability to formulate research questions and design experiments.
Strong quantitative and analytical reasoning.
Experience analyzing experimental results and drawing actionable conclusions.
Ability to develop new methodologies rather than only applying existing benchmarks.
Strong preference for applied research with practical product or engineering impact.
Strong written and verbal communication.
Comfortable giving and receiving constructive feedback.
Strong analytical and problem-solving mindset.
Highly curious about emerging AI technologies.
Comfortable working in ambiguous research environments.
Able to communicate complex technical concepts clearly.
Collaborative and team-oriented.
Comfortable working closely with engineers and researchers.
Able to balance research quality with practical execution.
Highly self-directed and intellectually curious.
Motivated by real-world impact rather than research publication alone.

Nice To Haves

Academic publications in NLP, benchmarking, or related areas are a plus but are not required.
Strong candidates do not need to be focused on publishing papers as their primary career objective.

Responsibilities

Evaluate new AI and language models as they are released.
Design and build new benchmarks from scratch.
Develop evaluation methodologies for measuring model quality, reliability, and performance.
Improve automated evaluation methods for generated text.
Design experiments to understand model strengths, weaknesses, and failure modes.
Explore new approaches to benchmarking increasingly capable AI systems.
Translate research questions into practical evaluation frameworks.
Construct datasets for evaluating language models and AI systems.
Work with labelers to create high-quality evaluation datasets.
Develop benchmark specifications, annotation processes, and quality standards.
Write technical documentation and white papers describing methodologies and results.
Build tooling that enables repeatable and scalable model evaluation.
Analyze evaluation results and identify opportunities to improve benchmarks.
Work with engineering to implement and scale evaluation systems.
Collaborate closely with software engineers to productionize evaluation methodologies.
Build clean, maintainable Python code for research and evaluation workflows.
Work with deep learning frameworks such as PyTorch and TensorFlow.
Explore language modeling, diffusion models, and other modern AI/ML techniques.
Work with Python-based web infrastructure such as Django or Flask where relevant.
Leverage AWS infrastructure to run and scale evaluation workloads.
Translate experimental findings into robust technical systems.
Work with leading AI labs and enterprise customers to understand evaluation needs.
Translate customer and research requirements into benchmark and evaluation strategies.
Communicate technical findings clearly to both technical and non-technical stakeholders.
Incorporate feedback from engineers, researchers, customers, and other team members.
Participate in collaborative research and development efforts.
Help define how advanced AI systems should be tested before deployment in enterprise environments.