Data Scientist, Applied AI

Definitive Healthcare, US • Framingham, MA

16h • $111,000 - $167,000

About The Position

This individual contributor role supports the team’s practical generative AI work. Under the direction of senior team members, the Data Scientist implements and tests LLM, RAG, and agent-based workflows for assigned use cases, producing evidence that informs decisions about which approaches to use. The role works independently on defined analyses and implementation tasks, with guidance on project priorities and broader technical direction. It makes day-to-day decisions about experiments, test cases, and implementation details within agreed project goals, and recommends model and retrieval improvements based on results. Major architecture changes, sensitive-data concerns, significant cost implications, and production release decisions are escalated to senior team members or appropriate owners. The role influences assigned projects through implementation and evidence rather than owning evaluation strategy or technical direction across multiple projects.

Requirements

• Bachelor’s degree in a quantitative or technical field.
• At least 3 years of experience in data science, machine learning, applied AI, or a related field.
• Hands-on experience developing applications with LLMs and evaluating their outputs.
• Strong Python, data analysis, and problem-solving skills.
• Understanding of embeddings, semantic search, retrieval, RAG, and evaluation metrics.
• Experience with an agent framework such as PydanticAI, LangGraph, or a similar tool.
• Ability to design useful test cases, interpret evaluation results, and identify limitations.
• Familiarity with cloud data or ML platforms, Git, testing, and collaborative software development.
• Clear communication with technical and nontechnical colleagues.

Nice To Haves

• Experience with Databricks, evaluation platforms, production deployment, fine-tuning, distributed processing, or healthcare data.

Responsibilities

• Build and test LLM, RAG, and tool-using agent workflows for assigned use cases.
• Compare LLMs, embedding models, and retrieval methods against quality, reliability, latency, and cost using evaluation criteria and methods set by senior team members.
• Create and update representative test cases, benchmark datasets, and regression checks; report results and limitations.
• Develop and tune RAG pipelines, including embeddings, retrieval, reranking, and grounding, based on evaluation results.
• Review incorrect answers, retrieval misses, and workflow failures; implement and measure improvements.
• Use experimentation, tracing, and observability tools to help monitor systems and investigate performance changes.
• Work with engineering, product, and domain partners to clarify requirements, communicate findings, and deliver practical AI solutions.
• Recommend model and retrieval improvements based on results, and escalate major architecture changes, sensitive-data concerns, significant cost implications, and production release decisions to senior team members or appropriate owners.

Benefits

• Medical coverage
• Dental coverage
• Vision coverage
• Unlimited paid time off
• 401(k) plan with employer contribution
• Potential eligibility to participate in a company bonus or commission plan, depending on the position

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level