Prompt Engineer

Medfar • Montreal, QC

4h • Remote

About The Position

As a Prompt Engineer, you will play a central role in improving generative AI features for clinical and operational use. You will help evolve the quality of generations while participating in the structuring of prompting practices within the team. Specifically, you will be responsible for:
Auditing and re-architecting CoeurWay's monolithic prompts: Converting their current structure into a modular chain of targeted and maintainable prompts, improving robustness, debuggability, and adaptability;
Leading the selection and evaluation of language models: Evaluating and pairing models to specific prompting tasks, favoring a model-agnostic approach that reduces vendor dependency and optimizes quality and costs;
Ensuring the quality of AI-generated clinical outputs: Defining quality criteria, identifying failure modes, and driving continuous improvement through structured validation processes;
Evolving and realigning the existing evaluation framework: Improving the framework's coverage, which currently covers 7 quality rubrics for clinical notes, its automation potential, and its alignment with clinical reality;
Establishing a systematic approach to output validation: Identifying edge cases, implementing regression tests for prompt modifications, and documenting known limitations;
Designing and implementing new prompts and evaluation suites: Supporting new CoeurWay and MYLE EMR features with appropriate prompting and evaluation solutions;
Mentoring and ensuring the growth of the practice: Mentoring future prompt designers as the team grows and participating in hiring decisions.
Contributing to documentation and best practices: Establishing and maintaining internal documentation, standards, and best practices for prompt engineering at MEDFAR;
Staying abreast of the evolving LLM landscape: Tracking new models, prompting techniques, and evaluation methodologies, and sharing relevant findings with the team;
Developing a deep understanding of clinical workflows: Gaining expertise in healthcare provider needs and maintaining close relationships with client-facing and end-user teams;
Collaborating with the GenAIOps team: Ensuring prompts are production-ready—observable, versioned, and deployable via established CI/CD processes.

Requirements

• 5 years minimum of professional experience in AI/ML, product development, computational linguistics, or a related field
• 2 years minimum of hands-on experience in prompt engineering within production LLM systems, ideally in a B2B SaaS context or regulated industry
• Demonstrated experience in decomposing complex prompts into modular, chained architectures, including patterns such as prompt chaining, routing, reflection, or tool usage
• Experience in designing or significantly improving LLM evaluation frameworks—designing rubrics, inter-rater alignment, and failure mode analysis
• Strong familiarity with commercial APIs for advanced generative models (OpenAI, Anthropic, Google, etc.) and understanding of practical trade-offs between them
• Advanced proficiency in English, written and spoken
• Proficiency in French required due to frequent client interactions

Nice To Haves

• Experience in health IT, clinical documentation, or healthcare-related use cases
• Understanding of concepts related to clinical notes, their structure, and the relevance of information to include or exclude
• Experience with automated quality evaluation approaches for generations, including the use of LLM-based judges
• Experience in contexts where multiple models are used according to different needs or constraints
• Familiarity with orchestration, monitoring, or operationalization challenges of GenAI solutions
• Strong interest in creating standards, documentation, and sustainable practices in an emerging field
• Proficiency in Python or another programming language for building evaluation pipelines or prompting tools
• Experience with self-hosted or open-weight models

Responsibilities

• Auditing and re-architecting CoeurWay's monolithic prompts: Converting their current structure into a modular chain of targeted and maintainable prompts, improving robustness, debuggability, and adaptability
• Leading the selection and evaluation of language models: Evaluating and pairing models to specific prompting tasks, favoring a model-agnostic approach that reduces vendor dependency and optimizes quality and costs
• Ensuring the quality of AI-generated clinical outputs: Defining quality criteria, identifying failure modes, and driving continuous improvement through structured validation processes
• Evolving and realigning the existing evaluation framework: Improving the framework's coverage, which currently covers 7 quality rubrics for clinical notes, its automation potential, and its alignment with clinical reality
• Establishing a systematic approach to output validation: Identifying edge cases, implementing regression tests for prompt modifications, and documenting known limitations
• Designing and implementing new prompts and evaluation suites: Supporting new CoeurWay and MYLE EMR features with appropriate prompting and evaluation solutions
• Mentoring and ensuring the growth of the practice: Mentoring future prompt designers as the team grows and participating in hiring decisions
• Contributing to documentation and best practices: Establishing and maintaining internal documentation, standards, and best practices for prompt engineering at MEDFAR
• Staying abreast of the evolving LLM landscape: Tracking new models, prompting techniques, and evaluation methodologies, and sharing relevant findings with the team
• Developing a deep understanding of clinical workflows: Gaining expertise in healthcare provider needs and maintaining close relationships with client-facing and end-user teams
• Collaborating with the GenAIOps team: Ensuring prompts are production-ready—observable, versioned, and deployable via established CI/CD processes

Benefits

• Remote work and flexibility (work-life balance)
• RRSP contribution
• Comprehensive group insurance from day one
• Paid time off: 3 weeks + 1 between Christmas and New Year's
• Annual training allowance to support professional development
• Onboarding program to familiarize you with our environment and the digital health field
• Computer equipment provided, as well as other equipment if necessary
• Opportunities for internal growth (promotions, internal mobility)
• Support from a wellness and social committee, with initiatives promoting cohesion, mental health, and employee well-being
• A corporate culture focused on transparency, collaboration, and innovation
• Joining a dynamic and innovative environment where your work has a concrete and large-scale impact, contributing to modernizing healthcare both in Canada and internationally

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed