Head of Central Quality and Project Enablement

HumanSignal•San Francisco, CA

4d•$140,000 - $180,000•Hybrid

About The Position

HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery. We design and create datasets from scratch, recruit and manage the domain experts who evaluate model output, and run everything through our own platform, Label Studio, the open-source standard for data labeling and evaluation, used by over 1 million practitioners worldwide. We specialize in the operationally complex: real-world data collection, multimodal pipelines, and multi-step workflows. Advanced ML and AI teams use our enterprise platform to run their own data factories, and our services team to extend their reach where in-house capacity runs out. If you want to do work that materially shapes how the next generation of AI products gets built, we'd love to talk. Every Data Services engagement succeeds or fails on two things: whether the team doing the work understands the spec, and whether we can prove the output meets it. This role owns both. As Head of Quality & Project Enablement, you'll take each customer spec and turn it into the materials that get annotators and experts productive. You'll then design the quality pipeline and sampling methodology that verifies the work before it ships. When quality slips, you'll trace it back to its source, whether that's a gap in the spec, the training, or the review process, and fix it.

Requirements

6+ years in quality assurance or quality operations for data labeling, human data, or ML training/evaluation data, including experience leading a quality function or team
A proven ability to translate complex or ambiguous specs into guidelines and training that produce consistent results
Strong applied statistics for QA: sampling design, confidence intervals, and agreement metrics, plus the ability to explain your choices to a customer
Hands-on experience configuring review and QA workflows in an annotation platform
Proficiency in Claude Code for quality analysis
Crisp written communication. Your guidelines and quality reports are the product.

Nice To Haves

Experience with LLM, RLHF, or preference-data projects
Experience with expert or domain-specialist workforces
A background in instructional design
Familiarity with Label Studio
Experience with model-assisted QA

Responsibilities

Turn customer spec documents into project-ready enablement packages: annotator guidelines, decision trees, edge-case libraries with worked examples, and onboarding walkthroughs.
Build qualification tests and gold-standard sets that confirm annotators understand the spec before they touch production work.
Run pilot and calibration rounds at project kickoff to surface spec ambiguities early. Resolve those ambiguities with the customer and delivery lead before scaling up.
Maintain versioned guidelines throughout the project. Roll out updates as new edge cases appear, and confirm the workforce has absorbed the changes.
Design the quality plan for every project. Choose the review structure (gold tasks, overlap/consensus, multi-stage review, expert adjudication) based on the task type, risk, and budget.
Set the sampling methodology. Size samples to hit target confidence levels, stratify by class and difficulty, and adjust review rates for each annotator based on performance.
Select the right metrics for each task, such as accuracy against gold, agreement measures like κ or α, or per-class error rates. Set acceptance thresholds that fit how subjective the task is.
Configure these workflows in our labeling platform, and turn what works into reusable quality playbooks by task type.
Own final quality sign-off. No delivery ships without meeting its agreed acceptance criteria.
Monitor quality throughout each project. Catch drift early, run root-cause analysis on defects, and drive corrective action for individuals and for guidelines.
Produce a clear quality report for every delivery that shows the methodology, the results, and any known limitations.