Strategic Projects Lead

Contra•New York, NY

1d•$150,000 - $200,000•Onsite

About The Position

Contra Labs is a human-centered research lab focused on understanding how creatives use AI, how AI tools can better align with creative needs, and how to preserve the diversity and originality that defines great creative work in an era of increasing homogenization. We connect a global network of top creative talent with frontier AI labs and product companies. By grounding our research in real creative workflows, we help define what "good" looks like across design, video, imagery, and beyond. Our work powers benchmarking, evaluation, and post-training for leading AI models and applications. Built on top of Contra, the professional network for independent creatives, Contra Labs turns human judgment into the training and evaluation layer that helps AI tools better serve the people who use them. Creatives are at the center of everything we do. Contra Labs is scaling across client engagements including post-training dataset creation, benchmarking programs, and ongoing evaluation partnerships. We need someone who can own the research and delivery side of this work end-to-end. Projects span AI labs, creative tool companies, and research teams, each with different requirements, timelines, and quality bars. This person scopes the methodology, designs the evaluation protocols, spins up the right evaluators, manages quality throughout, and makes sure we deliver rigorous, actionable results.

Requirements

Master's degree or PhD in HCI, cognitive science, psychology, design research, or a related field (or equivalent depth of industry experience)
3+ years of hands-on experience conducting research: designing studies, running evaluations, analyzing qualitative and quantitative data, and presenting findings
Experience with statistical methods including inter-rater reliability, sampling, regression analysis, and significance testing
Proficiency with data analysis tools (Python, SQL) for working with evaluation data at scale
Experience in data labeling, annotation, human-in-the-loop systems, or AI/ML evaluation workflows
Track record managing multiple concurrent projects, including annotation, evaluation, or QA workflows
Strong understanding of creative domains and what "good" looks like
Based in NYC, USA
In-office 5 days/week (Williamsburg)

Nice To Haves

Experience with RLHF, LLM evaluation, benchmark design, or human-in-the-loop systems
Background in ethnography, UX research, or design study methodologies
Startup or 0-to-1 experience

Responsibilities

Design and lead research studies including annotation protocols, rubrics, and evaluation methodologies for creative and multimodal outputs
Conduct and oversee qualitative and quantitative analysis, synthesizing findings into actionable insights and recommendations for clients
Interface with creative domain experts, manage evaluator teams, and calibrate for consistent quality through real-time course-correction
Develop and standardize annotation, data quality, and inter-rater reliability protocols across projects
Communicate research findings effectively to stakeholders through reports, presentations, and workshops
Collaborate with engineering on tooling needs and contribute to Contra Labs' positioning through case studies and benchmark publications
Translate client goals and priorities into well-scoped research plans, managing multiple concurrent projects with different timelines and requirements