About The Position

The Senior Data Scientist owns the diagnostic and evaluation loop for LLM-based clinical services. The role investigates failure modes surfaced by clinical reviewers and product managers, develops root-cause hypotheses across prompts, context assembly, retrieval, guideline encoding, model behavior, and upstream data, and designs experiments to isolate causes. It tests candidate fixes and verifies improvements through structured evaluations while checking for regressions. The role owns the evaluation roadmap and quality bar for the team's AI services, including deployment-gating metrics, the growth of golden sets and regression suites, and evidence for determining when a service is ready to ship. It extends the existing evaluation platform rather than rebuilding it, and partners with medical directors and product managers to turn review findings into labeled evidence and prioritized evaluation criteria. The position also mentors others and may take on direct reports as the team expands. It supports the annual clinical-guidelines update cycle through regression evaluation as guidelines, prompts, and models change, ramping with the team's senior data scientists in Q4 2026. The role documents a diagnostic playbook that others can use and handles clinical data, including PHI, in accordance with organizational security, privacy, and compliance requirements.

Requirements

  • Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field, or equivalent experience.
  • At least 5 years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production.
  • At least 2 years of recent, hands-on experience evaluating and improving LLM-based systems, including structured error analysis, prompt or configuration iteration, experiment design, and metrics interpretation.
  • Demonstrated experience with LLM evaluation methods and tools, including golden or regression sets, validated LLM-as-judge methods, and tracing and observability tooling.
  • Strong Python and solid data-analysis skills; SQL is a plus.
  • Comfort computing and reasoning about metrics such as sensitivity, specificity, and PPV.
  • Hypothesis-driven working style, with the ability to isolate variables and verify fixes.
  • Strong written communication skills to make findings and go/no-go evidence clear to engineers, clinicians, and leadership.

Nice To Haves

  • Healthcare experience in utilization management, prior authorization, clinical documentation, or clinical/claims data, and comfort reading clinical guidelines and medical-policy content.
  • Experience mentoring data scientists or leading small technical workstreams; interest in growing into people leadership as the function scales.
  • Experience partnering with clinical reviewers or other domain experts to convert expert judgment into labeled data and evaluation criteria.
  • Experience with LLM observability or telemetry stacks such as OpenTelemetry-based tracing, Logfire, Langfuse, or similar.
  • Statistics or experimentation background, including A/B testing, statistical significance, and sample-size reasoning.
  • Master's degree in a quantitative field.

Responsibilities

  • Investigate failure modes in LLM-based clinical services and develop root-cause hypotheses across prompts, context assembly, retrieval, guideline encoding, model behavior, and upstream data.
  • Design experiments to isolate causes and test candidate prompt, configuration, context, and model changes.
  • Verify improvements using structured evaluations and confirm that fixes do not regress other behavior.
  • Own the evaluation roadmap and quality bar, including deployment-gating metrics, golden sets, regression suites, and evidence-based shipping criteria.
  • Extend and use the team's evaluation platform; build evaluation sets, define metrics, and run and interpret evaluation batteries.
  • Partner with clinical reviewers to convert review findings into labeled evidence and executable evaluation criteria.
  • Partner with product managers to prioritize consequential failure modes.
  • Mentor others in evaluation methods and help grow the evaluation function, with readiness to take direct reports as the team expands.
  • Support annual clinical-guidelines updates with regression evaluation as guidelines, prompts, and models change; ramp with senior data scientists in Q4 2026.
  • Document failure taxonomies, experiment templates, and variant history in a diagnostic playbook others can use.
  • Handle clinical data, including PHI, according to organizational security, privacy, and compliance requirements.

Benefits

  • Comprehensive benefits, including health insurance benefits, for qualifying employees.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service