Strategic Project Lead

Contra•San Francisco, CA

22h•Onsite

About The Position

Contra Labs is a human-centered research lab studying how creatives actually use AI, how tools can serve creative work better, and how to protect the originality that gets flattened as everything starts to look the same. We connect a global network of top creative talent with frontier AI labs and product companies. By grounding our research in real creative workflows, we help define what "good" looks like across design, video, imagery, and beyond. Our work powers benchmarking, evaluation, and post-training for leading models and applications. Built on Contra, the professional network for independent creatives, we turn human judgment into the training and evaluation layer that helps these tools serve the people who use them. This role exists because we are scaling across client engagements: post-training dataset creation, benchmarking programs, and ongoing evaluation partnerships. Each project spans a different mix of AI labs, creative-tool companies, and research teams, with different requirements, timelines, and quality bars. We need someone to own this work from end to end. The Strategic Project Lead will act as the operator for a project or vertical, sitting between the client who needs the data, the creative experts producing it, and the internal engineering team building the tooling. The goal is to make all three groups feel like they are working on the same thing, and to ship results that are both rigorous and genuinely useful.

Requirements

3+ years of experience designing studies, running evaluations, analyzing data, and presenting findings.
Experience with data labeling, annotation, human-in-the-loop systems, or AI and ML evaluation workflows.
A track record of managing several concurrent projects, including annotation, evaluation, or QA work.
A strong point of view on creative quality and what "good" looks like.
High ownership, low ego, and a bias for action.
Ability to ship a v1, learn, and do what it takes to deliver.
Based in San Francisco, California.

Nice To Haves

Strong research background, both qualitative and quantitative.
Master's or PhD in HCI, cognitive science, psychology, design research, or a related field, or equivalent industry depth.
Familiarity with statistical methods for evaluation work: inter-rater reliability, sampling, regression, significance testing.
Experience with RLHF, LLM evaluation, or benchmark design.
Experience with ethnography, UX research, or design-study methods.
Startup or 0-to-1 experience.

Responsibilities

Define the methodology, including annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs.
Manage delivery, including scope, evaluator staffing, quality routing, timelines, and the client relationship, from kickoff to final readout.
Ensure quality of data, inter-rater reliability, and implement real-time course correction when necessary.
Perform qualitative and quantitative analysis and translate findings into clear recommendations, reports, and workshops that clients can act on.
Develop the systems behind the work, turning repeated tasks into clean, measurable processes, and partnering with engineering on tooling.
Contribute to the narrative by creating case studies and benchmark publications that enhance Contra Labs' market understanding.