Technical Program Manager - Frontier AI Data Labeling

Hippocratic AI•Menlo Park, CA

About The Position

Hippocratic AI is hiring a Technical Program Manager to run the human-in-the-loop data labeling engine behind its frontier healthcare AI. The role spans audio and conversation annotation, clinician-labeled training data, preference and RLHF data, and evaluations that inform release decisions. The data produced feeds directly into model training and evaluation loops, with close daily collaboration with ML researchers, applied scientists, and licensed clinicians. The position owns end-to-end delivery of labeling and annotation programs, from scoping and task design through execution, quality assurance, and delivery on live model-development cycles. It translates evaluation and training-signal needs into labeling tasks, rubrics, and success criteria; develops and scales an expert rater workforce; authors and iterates guidelines for high-stakes clinical judgment; and engineers data-quality processes. The role also tracks throughput, cost, and quality, manages vendor relationships, selects labeling and QA tools, and coordinates dependencies and timelines across Engineering, Clinical, and Infrastructure. By day 90, the successful candidate is expected to own a live labeling or evaluation program through a major delivery milestone, clear a critical pipeline bottleneck, and deliver a measurable quality or throughput improvement. Within 12 months, they are expected to run multiple concurrent programs, maintain a calibrated rater bench trusted by ML teams, and trace model improvements to delivered data.

Requirements

At least 2 years of technical program management or data operations experience in an AI/ML or data-intensive environment; candidates with 3 or more years are considered for exceptional experience.
At least 1 year of hands-on experience with data labeling, annotation, or model evaluation, including writing guidelines and managing quality at scale.
Experience sourcing and managing contractor or expert annotator workforces through marketplaces or labeling vendors.
Working knowledge of how annotation quality affects model training, including preference data, pairwise comparisons, and rater calibration (RLHF).
Strong understanding of quality metrics, including inter-annotator agreement (IAA), precision and recall on gold sets, and throughput-versus-accuracy tradeoffs.
Track record of owning critical data or evaluation projects under time pressure, working with ML engineers and applied scientists, and driving results without direct authority.
Comfort with ambiguity in a fast-moving, healthcare-adjacent AI environment.

Nice To Haves

Experience running or supporting RLHF or other training-data pipelines at an AI lab or applied ML team.
Clinical, healthcare, or other regulated, high-stakes data experience.
Familiarity with Label Studio, Scale AI, Surge AI, or SageMaker Ground Truth.
Experience qualifying specialized raters such as clinicians or bilingual subject-matter experts.
Exposure to reward hacking and specification gaming, and how annotation design can mitigate them.

Responsibilities

Own end-to-end delivery of labeling and annotation programs, including audio and voice-conversation labeling, supervised labeling, and preference and RLHF data.
Manage program scoping, task design, execution, quality assurance, and delivery against live model-development cycles.
Work with ML and Applied Science teams to turn evaluation and training-signal needs into labeling tasks, rubrics, and success criteria.
Build and scale an expert rater workforce by sourcing, screening, and onboarding clinicians and specialized raters through marketplaces, vendors, and in-house teams.
Design qualification tasks, set rates, and scale rater pools as program needs change.
Author and iterate labeling guidelines and annotation rubrics, particularly for subjective, high-stakes clinical judgment.
Design quality-control systems using gold sets, calibration sessions, spot checks, and inter-rater agreement tracking.
Detect rater drift and annotation bottlenecks before they affect training data.
Track throughput, cost, and quality.
Manage vendor relationships, including statements of work and performance reviews.
Choose labeling and QA tooling.
Coordinate dependencies and timelines across Engineering, Clinical, and Infrastructure.
Own live labeling or evaluation programs through major delivery milestones and deliver measurable quality or throughput improvements.
Run multiple concurrent labeling and evaluation programs and help ML teams trace model improvements to delivered data.