Member of Technical Staff - Evaluations

Liquid AI•San Francisco, CA

1d•Hybrid

About The Position

Liquid AI is seeking a Member of Technical Staff to work on evaluations within its Post-Training team. The role focuses on building reliable, reproducible, and cost-efficient internal evaluations that track real-world model behavior and inform post-training decisions. The work spans the internal evaluation suite and its infrastructure, in close collaboration with post-training researchers and teams including audio. A particular focus is evaluating small edge models, for which many public evaluations designed for frontier models are not suitable. The role includes designing held-out evaluations for capabilities such as agentic workflows, computer use, and cybersecurity; creating evaluations tailored to small edge models; and inspecting generated outputs to investigate unexpected scores. The person will also translate customer and open-source feedback into environments usable for data, training, and evaluation, and collaborate with other teams on their evaluation needs. The position offers ownership of the evaluation function alongside one engineer, compute for running evaluations at scale, and opportunities to release open-source evaluations, publish papers, and report on failure modes in popular evaluations. Success in the first year includes internal evaluations covering major target capabilities and informing post-training decisions, a reliable suite co-owned across teams, and at least one open-source evaluation released with a paper, along with a failure-mode report on a popular evaluation.

Requirements

Strong evaluation-design ability: translate a high-level capability description into an accurate, general-purpose evaluation that distinguishes stronger from weaker models and is difficult to game.
Demonstrable prior evaluation work that can be reviewed and that holds up in coverage, reproducibility, contamination handling, and intended signal; at least one project must have been built from scratch, such as a benchmark or evaluation paper, or substantial contributions to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.
Fluency with evaluation fundamentals, including contamination, saturation, judge bias, and variance.
Hands-on experience running and managing evaluations in a team, such as on an evaluation or training team, or in an engineering/DevOps role.
Habit of examining individual generated answers to debug unexpected scores before accepting the numerical result.
Strong software engineering skills to own evaluation infrastructure and debug inference and infrastructure failures encountered when running evaluations at scale without hand-holding.
Proactive execution: independently pick up use cases, deliver working benchmarks, make progress visible, parallelize work, and finish reliably.
Openness about evaluation limitations and mistakes, including communicating when evaluations or scores were wrong and presenting numbers with their caveats.

Nice To Haves

Model training experience.
Publications or a research background; industry experience is equally welcome.
Deep systems expertise.
Familiarity with the strengths and limitations of small edge models.

Responsibilities

Maintain, optimize, and ensure the reliability and reproducibility of internal evaluation infrastructure at scale.
Design held-out evaluations for priority capabilities such as agentic workflows, computer use, and cybersecurity, documenting how they were constructed and checking for contamination.
Build evaluations targeted at small edge models.
Inspect individual generated outputs to debug unexpected evaluation scores before decisions are made based on them.
Turn customer and open-source feedback into environments usable for data, training, and evaluation.
Partner with other teams to design evaluations for their specific needs.
Release open-source evaluations, write papers about them, and publish failure-mode reviews of popular evaluations.
Help ensure internal evaluations cover target capabilities and inform post-training decisions; co-own the evaluation suite with other teams.