Senior/Staff Applied AI Engineer

Tali AI•Toronto, ON

1d•CA$170,000 - CA$230,000

About The Position

Tali AI is seeking a Senior/Staff Applied AI Engineer to own AI systems end-to-end, including models, prompts, retrieval, data, evaluations, and debugging. The role involves working on an ambient AI scribe that listens to visits and writes clinical notes, a recommender system for billing codes, medical search for clinical questions, and various other agents. A strong emphasis is placed on evaluation as a critical product feature to ensure trustworthiness. The engineer will be responsible for building evaluation pipelines, diagnosing and fixing failure modes, developing production agents with tool use and orchestration, debugging end-to-end visit issues, managing model routing and rollouts, and owning these systems in production. This role also involves translating clinical feedback into actionable plans and setting the standard for applied AI at Tali.

Requirements

5+ years in production ML, applied AI, or research engineering with experience owning systems for real users.
Deep evaluation experience, including building graders, regression suites, or judge pipelines.
Experience with agentic systems involving multiple models, tool calls, retrieval, and failure recovery.
Strong systems engineering skills, including backend services, data pipelines, and observability.
Data-centric approach to improving AI systems through data and feedback loops.
Proficiency in Python and modern ML tooling.
Ability to provide and receive candid feedback.
Skill in making engineering and business cases for technical decisions.
Ability to raise the skill level of surrounding engineers through mentorship and code review.

Nice To Haves

Speech recognition or real-time audio experience.
Experience in a regulated domain like healthcare or finance.
Clinical experience of any kind.

Responsibilities

Build evaluation pipelines that determine if AI is suitable for shipping, including automated judges, regression suites, human review, and underlying datasets.
Diagnose failure modes in AI systems and implement fixes, which may involve prompts, retrieval, routing, models, audio capture, or fine-tuning.
Build production agents, including tool use, orchestration, guardrails, and recovery mechanisms.
Develop supporting infrastructure for agents, such as search, vector storage, and agent harnesses.
Debug individual visits end-to-end, identify similar cases, and prove fixes.
Determine and safely change which model serves which request, using weighted routing and staged rollouts.
Own AI systems in production, responding to alerts, identifying causes of quality slips, and deciding on fixes.
Translate vague clinical complaints into problem statements, metrics, and actionable plans.
Set the standard for applied AI at Tali through the development of evaluation tools and processes.