Computational Linguist, AI Evaluation

TwelveLabs•San Francisco, CA

1d•$140,000 - $160,000•Hybrid

About The Position

TwelveLabs is seeking a Computational Linguist for their AI Evaluation team. This role is crucial for defining and executing data collection, labeling operations, and model quality measurement for multimodal AI models. The position involves high ownership, including defining dataset and evaluation requirements, designing data pipelines, coordinating with vendor partners, and automating repetitive tasks. Success requires cross-functional collaboration and strong relationship-building skills. The role is hybrid in San Francisco, with onsite work on Tuesdays and Thursdays.

Requirements

Direct experience building or running model evaluation pipelines (benchmark design, human eval frameworks, automated scoring systems), and translating ambiguous quality questions into measurable criteria.
5+ years of experience working in an AI focused data operations organization.
Proven track record designing and executing large scale data projects, including gathering, labeling, and post-processing data.
Ability to analyze messy and complex data, identify overarching patterns, and distill findings into crisp annotation guidelines or other accessible documentation.
Proficiency with Python, agentic coding, or other popular industry tools for automation.
Excellent communication and project management skills, and the ability to support several projects simultaneously.
Foundational understanding of and interest in LLMs/VLMs and multimodal AI.
Conviction that data is the key ingredient for the performance of AI models.

Nice To Haves

Experience in data collection and labeling for multimodal language models.
Experience working with research scientists and engineers.
Experience planning and operating new data tools and labeling systems.
Expertise or interest in video-centric domains, such as sports, advertising, and content creation.

Responsibilities

Define and build evaluation protocols for video-language model quality, translating ambiguous quality questions into measurable, repeatable benchmarks or evaluation flows in consultation with Research and Product teams.
Build pipelines and conduct analyses to transform raw customer and usage data into structured signals, identifying areas where models underperform and collaborating with cross-functional partners to develop improvement plans.
Design and execute video-language data collection and labeling projects, automating repetitive processes to allow the team to focus on higher-leverage work.
Coordinate with vendor and outsourcing partners executing at scale, maintaining high quality through clear guidelines and feedback loops.
Collaborate closely with Engineering and Research teams to align on priority data and evaluation needs, communicating findings through dashboards and reports to drive decisions.