Data Scientist, Applied AI

Definitive Healthcare • Framingham, MA

1d

About The Position

The Data Scientist, Applied AI supports practical generative AI applications by implementing and testing LLM, RAG, and agent-based workflows for assigned use cases under the direction of senior team members. The role produces evidence to inform team decisions about which approaches to use. It is an individual contributor position with no people-management responsibilities. The Data Scientist works independently on defined analyses and implementation tasks, with guidance on project priorities and broader technical direction. They make day-to-day decisions about experiments, test cases, and implementation details within agreed project goals, and recommend model and retrieval improvements based on results. Major architecture changes, sensitive-data concerns, significant cost implications, and production release decisions are escalated to senior team members or appropriate owners.

Definitive Healthcare turns data, analytics, and expertise into intelligence for healthcare customers. The company operates across North America, Europe, and India and supports a global client base of more than 2,400 customers. The company describes its culture as collaborative and inclusive, with programs including DefinitiveCares and employee-led affinity groups.

Requirements

• Bachelor’s degree in a quantitative or technical field.
• At least 3 years of experience in data science, machine learning, applied AI, or a related field.
• Hands-on experience developing applications with LLMs and evaluating their outputs.
• Strong Python, data analysis, and problem-solving skills.
• Understanding of embeddings, semantic search, retrieval, RAG, and evaluation metrics.
• Experience with an agent framework such as PydanticAI, LangGraph, or a similar tool.
• Ability to design useful test cases, interpret evaluation results, and identify limitations.
• Familiarity with cloud data or ML platforms, Git, testing, and collaborative software development.
• Clear communication with technical and nontechnical colleagues.

Nice To Haves

• Experience with Databricks.
• Experience with evaluation platforms.
• Experience with production deployment.
• Experience with fine-tuning.
• Experience with distributed processing.
• Experience with healthcare data.

Responsibilities

• Build and test LLM, RAG, and tool-using agent workflows for assigned use cases.
• Compare LLMs, embedding models, and retrieval methods for quality, reliability, latency, and cost using evaluation criteria and methods set by senior team members.
• Create and update representative test cases, benchmark datasets, and regression checks, and report results and limitations.
• Develop and tune RAG pipelines, including embeddings, retrieval, reranking, and grounding, based on evaluation results.
• Review incorrect answers, retrieval misses, and workflow failures; implement and measure improvements.
• Use experimentation, tracing, and observability tools to help monitor systems and investigate performance changes.
• Work with engineering, product, and domain partners to clarify requirements, communicate findings, and deliver practical AI solutions.
• Escalate major architecture changes, sensitive-data concerns, significant cost implications, and production release decisions to senior team members or appropriate owners.

Benefits

• Medical coverage.
• Dental coverage.
• Vision coverage.
• Unlimited paid time off.
• Participation in the company’s 401(k) plan with employer contribution.
• Depending on the position, eligibility to participate in a company bonus or commission plan.
• DefinitiveCares community service group.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level