Senior Data Scientist, Applied AI

Definitive Healthcare • Framingham, MA

1d

About The Position

The Senior Data Scientist, Applied AI independently owns the development and evaluation of AI solutions for an assigned use case, working within the team’s established evaluation framework and practices. The role applies professional judgment to technical approaches, issue diagnosis, and implementation within the assigned project, while escalating decisions that affect the team’s overall evaluation strategy, span multiple projects, or carry material risk. Performance is measured by the quality and reliability of the systems and evidence delivered for the assigned use case.

This is an individual contributor position with no people-management responsibilities. The Senior Data Scientist collaborates with immediate engineering, product, and domain partners and escalates matters involving changes to the team’s evaluation approach, significant business risk, sensitive data, material cost commitments, or production release approval to the Manager or appropriate owner.

Requirements

• Bachelor’s degree required.
• At least 5 years of relevant experience in data science, machine learning, applied AI, software engineering, or a related field.
• Hands-on experience building and evaluating LLM applications.
• Strong Python programming, software engineering and testing, and analytical problem-solving skills.
• Strong knowledge of LLMs, embeddings, semantic search, retrieval systems, and RAG architectures.
• Strong hands-on experience with Databricks, including Delta Lake, Unity Catalog, Lakebase, Lakeflow Jobs, and MLflow, for building and deploying scalable, production-grade data and AI solutions.
• Ability to design representative evaluations for an assigned use case, analyze errors, and use results to improve a system.
• Ability to weigh quality, reliability, latency, and cost when recommending an approach for an assigned project.
• Clear communication and effective collaboration with technical and nontechnical partners.
• Ability to work independently on an assigned project within the team’s established evaluation framework and practices.

Nice To Haves

• Master’s degree in a quantitative or technical field.
• Experience with agent frameworks and AI evaluation platforms.
• Experience with healthcare data.

Responsibilities

• Independently build, test, and improve LLM, RAG, and tool-using agent workflows for an assigned use case using maintainable, tested Python code.
• Select and apply evaluation criteria, baselines, and test datasets within the team’s established evaluation approach.
• Compare models, embeddings, and retrieval methods for the assigned use case using evidence about quality, reliability, latency, and cost; document project recommendations and tradeoffs.
• Investigate retrieval misses, incorrect answers, and workflow failures within the assigned project; determine root causes and implement and validate fixes.
• Build and update test cases, benchmark datasets, and regression checks for the assigned use case.
• Monitor and investigate performance changes for owned workflows and recommend and implement improvements within the project.
• Communicate findings and technical tradeoffs to immediate engineering, product, and domain partners.
• Escalate decisions involving changes to the team’s evaluation approach, significant business risk, sensitive data, material cost commitments, or production release approval to the Manager or appropriate owner.

Benefits

• Medical coverage
• Dental coverage
• Vision coverage
• Unlimited paid time off
• 401(k) plan with employer contribution
• Employees may be eligible for a company bonus or commission plan, depending on the position

What This Job Offers

• Job Type: Full-time
• Career Level: Senior