Senior AI Scientist

IQVIA•Durham, NC

15h•$108,000 - $270,000

About The Position

We are seeking a Senior AI Scientist to lead the design, development, and operationalization of evaluation frameworks for Generative AI systems, with a primary focus on Large Language Models (LLMs) and agentic AI solutions. This role will be responsible for defining and implementing robust methods to assess quality, safety, reliability, and business impact across LLM-powered applications and multi-agent workflows. The position operates within regulated environments such as life sciences, clinical research, and regulatory domains, ensuring that AI systems meet enterprise and compliance standards.

Requirements

Master’s or PhD in Computer Science, Artificial Intelligence, Machine Learning, or a related field
5+ years of experience in applied AI/ML, with a strong focus on Generative AI and LLMs
Demonstrated experience in: LLM evaluation (both automated and human-in-the-loop); Prompt engineering and model behavior analysis; Python programming using frameworks such as PyTorch or TensorFlow
Hands-on experience with: RAG systems, embeddings, and vector databases; Agent frameworks (e.g., LangChain, LangGraph, Semantic Kernel)
Strong understanding of: Evaluation metrics and experimental design; Model limitations, failure modes, and debugging techniques

Nice To Haves

Experience in life sciences, healthcare, or regulated environments
Familiarity with: Clinical trial workflows (ICF, CSR, TMF, regulatory submissions); Knowledge graphs and biomedical data systems
Experience with evaluation tools such as LangSmith, Promptfoo, DeepEval, HELM, or OpenAI Evals
Exposure to Responsible AI frameworks and regulatory compliance standards

Responsibilities

LLM Evaluation & Benchmarking: Design and implement scalable evaluation frameworks for LLMs across use cases including: Question answering, summarization, information extraction, and reasoning; Clinical and regulatory document generation (e.g., ICFs, CSRs, protocols). Develop both automated and human-in-the-loop evaluation pipelines. Define, measure, and monitor key performance metrics, including: Accuracy, factuality, faithfulness, and hallucination rate; Robustness, consistency, latency, and cost-performance trade-offs. Build domain-specific benchmarks using real-world clinical, regulatory, and RWD data.
Agentic AI & Multi-Agent Evaluation: Establish evaluation strategies for agent-based and multi-agent systems. Measure and analyze: Task completion success rates; Planning and reasoning quality; Tool usage accuracy; Inter-agent coordination and failure patterns. Develop scenario-based and simulation-driven evaluation environments. Evaluate orchestration frameworks (e.g., LangGraph, Semantic Kernel, Claude Agents).
End-to-End System Evaluation: Define evaluation strategies for complete AI pipelines, including: Retrieval-Augmented Generation (RAG) systems; Tool-augmented agents; Knowledge graph + LLM architectures. Implement offline and online evaluation mechanisms, such as: A/B testing and canary releases; Production monitoring and model drift detection. Enable observability and traceability using tools such as LangSmith and OpenTelemetry.
Responsible AI & Compliance: Ensure all evaluation practices align with Responsible AI principles and regulatory requirements (e.g., GxP). Assess and mitigate risks related to: Bias, fairness, safety, and explainability; Data leakage and privacy concerns. Develop audit-ready evaluation frameworks suitable for regulated environments (e.g., FDA, EMA).
Tooling & Platform Development: Build and scale evaluation tooling, including: Automated evaluation pipelines; Prompt and version tracking systems; Experiment management platforms. Integrate evaluation frameworks with enterprise AI platforms (e.g., Azure, Databricks, AWS, on-prem GPU environments).
Leadership & Collaboration: Collaborate with cross-functional teams including: AI/ML engineers, product teams, and domain scientists; Clinical, regulatory, and real-world evidence stakeholders. Establish enterprise-wide evaluation standards and best practices. Mentor junior team members and contribute to strategic AI initiatives.