Data Scientist

Accuity

26d•Remote

About The Position

The Data Scientist develops, evaluates, and monitors the predictive and generative AI systems that drive automation across Accuity’s clinical documentation and revenue cycle workflows. Reporting to the Vice President, Data Science and AI, this role combines classical machine learning with applied work on large language model and agent-based systems, and is accountable for establishing whether those systems are measurably good enough to be trusted inside a clinical workflow. Evaluation is the spine of the role. The Data Scientist builds and maintains the ground truth datasets, evaluation frameworks, and metrics that determine production readiness, designs samples so results are comparable and defensible, and analyzes performance by clinical and payer segment rather than in aggregate. This includes independently reproducing and pressure-testing results reported by external AI development partners rather than accepting them as delivered. This role owns methodology, evaluation, and model development. Production deployment, serving infrastructure, and the engineering of AI systems into live workflows are owned by AI Engineering, and the Data Scientist partners closely with that function to move validated work into production.

Requirements

Bachelor’s degree in computer science, statistics, mathematics, engineering, data science, or a related quantitative field required.
4+ years of applied data science, machine learning, or applied AI experience, including work that reached production.
Demonstrated experience building and evaluating models that were actually used to make decisions, rather than research or coursework alone.
Experience with evaluation methodology, including ground truth construction, sampling design, and metric definition.
Strong proficiency in Python and SQL.
Healthcare data experience required.
Employees must be able to communicate via phone, email, etc. and sit for extended periods of time, with or without reasonable accommodations.

Nice To Haves

Master’s degree in a related field preferred.
Exposure to large language model or agent-based systems, including prompt design and systematic evaluation of non-deterministic output, preferred.
experience with clinical coding, DRG assignment, CDI, or claims and revenue cycle data strongly preferred.
Experience with working with protected health information in a HIPAA-regulated environment preferred.
Experience with cloud data and machine learning platforms, preferably Microsoft Azure and Databricks, preferred.

Responsibilities

Develop, train, and improve predictive and machine learning models supporting chart triage, prioritization, documentation integrity, and revenue cycle outcomes.
Maintain and improve models already running in production, including retraining cadence, feature review, threshold tuning, and recalibration as data and workflow change.
Identify and correct sampling and selection bias in training data, including bias introduced when a model’s own decisions determine which records are subsequently observed.
Contribute to applied work on large language model and agent-based systems, including prompt and workflow design, tool integration, and systematic failure mode analysis.
Build feedback loops so findings from clinical audit and production review flow back into model, threshold, and system decisions rather than stopping at a report.
Prototype and test new approaches, and state plainly and early when an approach does not work.
Build and maintain ground truth datasets, including sourcing, annotation coordination with clinical subject matter experts, quality assessment, and remediation of known data defects.
Design evaluation samples deliberately, understanding when a stratified or weighted sample is appropriate, when a near-production distribution is required, and when results across runs are not comparable.
Define, compute, and document the metrics that determine production readiness, and ensure another person can reproduce them from source data.
Analyze performance by clinically meaningful segment, including diagnosis group, service line, payer, and case complexity, rather than reporting aggregate accuracy alone.
Analyze error in both directions, distinguishing false positives from false negatives and quantifying the clinical and financial consequence of each.
Maintain evaluation tooling and harnesses so experiments are repeatable and every result is traceable to a specific dataset, configuration, and version.
Monitor deployed models and AI systems for accuracy, calibration, drift, and subgroup performance, and surface issues before they appear as business impact.
Maintain documentation of model design, assumptions, limitations, training data, and known failure modes to a standard that supports audit and clinical review.
Support AI governance requirements, including human-in-the-loop design, decision traceability, and evidence of validation.
Contribute to the definition and computation of metrics that carry commercial or contractual weight, ensuring they are reproducible and auditable.
Work directly with physicians, coders, and clinical subject matter experts to define correct outcomes, resolve disagreements about ground truth, and validate model behavior against real clinical judgment.
Translate clinical and operational problems into well-posed analytical problems, and translate results back into terms a clinical or business audience can act on.
Present findings, limitations, and recommendations honestly, including where results are inconclusive or where the available data will not support the question being asked.
Help the business understand what a model can and cannot be relied on to do.
Work alongside external AI development partners, independently reproducing and verifying reported results rather than accepting them as delivered.
Review partner methodology, sampling design, and metric definitions, and raise discrepancies clearly and early.
Absorb knowledge from partner-delivered systems so Accuity can evaluate, maintain, and improve them without ongoing external dependency.
Partner with AI Engineering to move validated models and systems into production, providing the specifications, evaluation criteria, and acceptance thresholds that define success.
Partner with Data Architecture on the data assets, structure, quality, and access required for model development and evaluation.
Write clean, reviewable, version-controlled code and work in shared repositories rather than personal notebooks.
Document work so another data scientist can reproduce it without the original author present.
Handle protected health information in accordance with HIPAA, HITRUST, and Accuity policy at all times.
Follow approved practice for data access, data movement, model routing, and any use of third-party services involving PHI.
Support audit readiness by producing accurate, timely evidence of validation and evaluation work.
Perform additional responsibilities as needed to support data science, technology, and organizational goals.