Technical Program Manager

Hippocratic AI•Menlo Park, CA

14d

About The Position

Hippocratic AI is hiring a Technical Program Manager to run the human-in-the-loop data labeling engine behind its frontier healthcare AI. The role covers audio and conversation annotation, clinician-labeled training data, preference and RLHF data, and evaluations used to determine what ships. The data produced feeds directly into model training and evaluation loops, with close daily collaboration with ML researchers, applied scientists, and licensed clinicians. The Technical Program Manager owns end-to-end delivery of labeling and annotation programs, including scoping, task design, execution, quality assurance, and delivery against active model-development cycles. The role turns evaluation and training-signal needs into labeling tasks, rubrics, and success criteria, and develops and scales a qualified workforce of clinicians and specialized raters. It also authors and iterates guidelines for high-stakes clinical judgment, engineers quality controls, monitors throughput, cost, and quality, manages vendor relationships and tooling, and aligns Engineering, Clinical, and Infrastructure teams on dependencies and timelines. By day 90, the person is expected to own a live labeling or evaluation program through a major delivery milestone, clear a critical pipeline bottleneck, and deliver a measurable quality or throughput improvement. Within 12 months, they are expected to run multiple concurrent programs, maintain a calibrated rater bench trusted by ML teams, and trace model improvements back to delivered data.

Requirements

At least 5 years in technical program management or data operations in an AI/ML or data-intensive environment; 3+ years may be considered for exceptional candidates.
At least 2 years of hands-on experience with data labeling, annotation, or model evaluation, including writing guidelines and managing quality at scale.
Experience sourcing and managing contractor or expert annotator workforces through marketplaces or labeling vendors.
Working knowledge of how annotation quality affects model training, including preference data, pairwise comparisons, and rater calibration (RLHF).
Strong grasp of quality metrics, including inter-annotator agreement, precision and recall on gold sets, and throughput-versus-accuracy tradeoffs.
Track record of owning critical data or evaluation projects under time pressure, working with ML engineers and applied scientists, and driving results without direct authority.
Comfort with ambiguity in a fast-moving, healthcare-adjacent AI environment.

Nice To Haves

Experience running or supporting RLHF or other training-data pipelines at an AI lab or applied ML team.
Clinical, healthcare, or other regulated, high-stakes data experience.
Familiarity with Label Studio, Scale AI, Surge AI, or SageMaker Ground Truth.
Experience qualifying specialized raters such as clinicians or bilingual subject-matter experts.
Exposure to reward hacking and specification gaming, and how annotation design can mitigate them.

Responsibilities

Own end-to-end delivery of labeling and annotation programs, including audio and voice-conversation labeling, supervised labeling, and preference and RLHF data.
Manage program scoping, task design, execution, quality assurance, and delivery against live model-development cycles.
Work with ML and Applied Science to translate evaluation and training-signal needs into labeling tasks, rubrics, and success criteria.
Develop evaluations and benchmarks that gate model releases.
Source, screen, and onboard clinicians and specialized raters through marketplaces, vendors, and in-house teams.
Design rater qualification tasks, set rates, and scale rater pools as programs change.
Author and iterate labeling guidelines and annotation rubrics, especially for subjective, high-stakes clinical judgment.
Design quality-control systems using gold sets, calibration sessions, spot checks, and inter-rater agreement tracking.
Identify rater drift and annotation bottlenecks before they affect training.
Track throughput, cost, and quality.
Manage vendor relationships, including statements of work and performance reviews.
Choose labeling and quality-assurance tooling.
Coordinate dependencies and timelines across Engineering, Clinical, and Infrastructure.
Own critical data or evaluation projects under time pressure and drive results without direct authority.