Applied Data Scientist, Health AI Evaluation & Datasets

Innodata Inc.

3h•$150,000 - $175,000

About The Position

Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide data, evaluation frameworks, and human expertise for building trustworthy AI systems at scale. Our mission is to support Generative AI/AI builders and adopters with transferable solutions, platforms, and services, building on our 36+ year legacy of delivering high-quality data and outstanding outcomes. Healthcare is a critical domain for generative AI, requiring clinical accuracy, patient safety, regulatory compliance, health equity, auditability, and workflow fit. Innodata collaborates with foundation model labs, medical AI startups, payers, providers, pharma, and digital health companies to build LLMs, multimodal systems, and AI agents for healthcare and life sciences. As an Applied Data Scientist, Health AI Evaluation & Datasets, you will be responsible for the design, measurement quality, and clinical validity of datasets used for training, fine-tuning, and evaluating health-domain models. This role requires a blend of clinical or biomedical fluency and data science rigor, enabling you to interpret clinical guidelines, payer policies, medical literature, and patient communication workflows, translate them into measurable datasets and evaluation plans, and effectively communicate your methodology to clinical, data science, and ML stakeholders. You will work collaboratively in a dedicated pod with a Technical Solutions Architect, Applied Research Scientist, AI/ML Research Engineer, and Language Data Scientists, ensuring that data, rubrics, review workflows, and measurement evidence are clinically realistic, statistically defensible, compliant, and useful for evaluation and post-training.

Requirements

5+ years of data science experience, with at least 2+ years in healthcare, clinical, biomedical, payer, provider, pharma, life sciences, or comparable regulated health data.
Working knowledge of healthcare data and standards (EHR structure, clinical documentation, ICD-10, CPT, SNOMED CT, LOINC, RxNorm, familiarity with FHIR, HL7).
Hands-on experience designing ML datasets, including writing annotation guidelines, sizing cohorts, setting quality thresholds, and designing QA checks.
Familiarity with LLM-based health AI workflows (prompt design, rubric-based evaluation, RAG, LLM-as-judge, model comparison).
Strong Python and SQL skills; comfort with pandas, scikit-learn, statsmodels or equivalent tools.
Working familiarity with modern LLM tooling (Hugging Face, evaluation frameworks, prompt development tools, model APIs).
Statistical literacy across sampling design, bias and fairness analysis, inter-annotator agreement metrics, confidence intervals, significance testing, and error analysis.
Solid grasp of healthcare privacy, compliance, and governance (HIPAA, de-identification standards, PHI handling, auditability, access control).
Ability to work credibly with clinicians, biomedical SMEs, research scientists, engineers, technical solutions teams, annotators, and customer stakeholders.
A bias toward clinical realism, prioritizing datasets that reflect real-world scenarios.
Degree in a relevant field such as biostatistics, epidemiology, computational biology, health informatics, computer science with a health focus, statistics, or a clinical degree with quantitative training, or equivalent demonstrated experience.

Nice To Haves

Clinical credentials (MD, RN, PharmD, MPH, PhD, or health informatics backgrounds are especially encouraged).

Responsibilities

Translate customer goals into dataset specifications, taxonomies, rubrics, sampling plans, and acceptance criteria for various health AI applications.
Focus on multimodal health AI, designing training and evaluation datasets across diverse data types (clinical text, medical images, waveforms, EHR data, claims, trial data, etc.) and use cases (clinical reasoning, medical QA, note summarization, etc.).
Design evaluations for retrieval-augmented and source-grounded health AI systems, assessing aspects like evidence citation, faithfulness, and guideline adherence.
Define sampling strategies, label schemas, inter-annotator agreement targets, adjudication workflows, SME review patterns, and quality thresholds in collaboration with various teams.
Build statistical and ML checks to ensure healthcare dataset trustworthiness, including bias analysis, leakage detection, and reliability metrics.
Partner with research scientists and engineers to integrate datasets into evaluation and post-training pipelines.
Evaluate health AI behavior beyond surface accuracy, considering calibration, hallucination, robustness, equity, and safe handoff.
Reason concretely about clinical workflow fit and the evidence needed to trust AI outputs in care delivery.
Own data quality from source intake through delivery, ensuring PHI/PII handling, provenance, audit trails, and compliance documentation.
Stay current on the health AI landscape, including regulatory developments and benchmark releases.
Support customer discovery and proposal work by scoping dataset programs and estimating effort.
Contribute to Innodata's internal IP by developing reusable health-domain assets like taxonomies, rubrics, and methodology templates.