Senior Scientist, Clinical AI Research & Evaluation (Regulatory & Responsible AI)

AstraZeneca•Cambridge, MA

6d•$124,714 - $187,070•Hybrid

About The Position

AstraZeneca is hiring a researcher-first Senior Scientist to advance the evaluation of AI systems used in clinical development and to provide technical advice to Regulatory Affairs. The role works with the Clinical AI Applied Science & Translation team on evaluations of LLM-based and agentic systems supporting clinical-trial planning, conduct, monitoring, and review. It also translates evidence about system reliability, responsible AI, limitations, uncertainty, and residual risk into actionable guidance for Regulatory colleagues. The scientist will develop reusable evaluation methods and assets, investigate how AI outputs affect clinical judgment, analyze system failures, and communicate findings. In partnership with Regulatory, Quality, and compliance colleagues, the role will help establish evidence and validation expectations early in system design. GxP expertise is not required; the role calls for a strong researcher interested in learning evidentiary standards for regulated medicine. The position is based in Cambridge, Massachusetts, with an expectation to work from the office an average of at least three days per week, while respecting individual flexibility.

Requirements

PhD in Machine Learning, Computer Science, NLP, Information Science, Biomedical Informatics, Statistics, or a related quantitative discipline; recent PhD through approximately three years post-PhD, or a master's degree with an equivalent research record.
At least three years of hands-on Python and SQL experience, including version control, tests, reproducible experiments, and turning research questions into clean, runnable code.
Demonstrated LLM or NLP evaluation experience, such as benchmarks, LLM-as-judge, error analysis, or metric selection, evidenced by publications, released systems, or open-source work.
A research track record demonstrated by peer-reviewed publications or equivalent evidence of rigorous, communicated science.
Ability to explain uncertainty and residual risk to non-specialists, including clinical, quality, and regulatory audiences, without overstating or obscuring.
Curiosity about clinical development and regulated medicine, and motivation to learn GxP, validation, and health-authority context on the job.

Nice To Haves

Clinical NLP experience with radiology reports, clinical notes, EHR, safety data, or biomedical language models.
Experience designing user studies or expert-agreement studies, and causal inference or statistical rigor beyond predictive accuracy.
Industry research-lab experience or equivalent experience shipping AI systems used by real users.
Exposure to regulated environments, including model risk management, audit readiness, validation plans, or health-authority frameworks.
Familiarity with evaluating retrieval-augmented or agentic systems, red-teaming, or safety evaluation.

Responsibilities

Design and run evaluations of LLM-based and agentic clinical systems, including benchmark construction, evaluation harnesses, calibrated LLM-as-judge methods, failure-mode taxonomies, red-teaming, and error analysis.
Build reusable evaluation assets, including test sets, scoring rubrics, annotation protocols, regression suites, and analysis pipelines.
Study how AI outputs shape clinical judgment through expert-agreement studies, user studies, and analysis of reliability when ground truth is noisy, incomplete, or context-dependent.
Investigate failures using traces, logs, qualitative review, and quantitative analysis, and translate findings into model, prompt, tool, data, or workflow improvements.
Publish and communicate research findings internally and externally where appropriate.
Translate evaluation evidence, residual risk, model limitations, and uncertainty into actionable language for Regulatory Affairs.
Advise on evidence standards for AI systems approaching regulated use, including validation documentation, traceability, audit trails, intended-use clarity, and human-oversight definition.
Track evolving health-authority expectations for AI in clinical decision support and regulated contexts, and brief the team and Regulatory colleagues on their implications.
Support validation-planning and risk-assessment discussions with Quality and Regulatory from early design.
Write production-quality Python and SQL for evaluation and analysis, maintaining reproducible, versioned, and tested experiments.
Partner with clinical subject-matter experts, product owners, and engineering to ground evaluations in real workflows and stakes.
Contribute to technical design reviews, code reviews, documentation, risk assessments, and release evidence.