Staff AI Engineer - QA

Harris Computer•New York, SC
•$120,000 - $160,000•Remote

About The Position

Harris Computer is seeking a Staff AI Engineer focused on quality assurance to lead the quality and reliability of AI and machine learning solutions for healthcare. The role is the technical authority for ensuring AI systems are robust, fair, accurate, and safe, and will help define testing standards for intelligent systems that can affect patient care. This full-time remote position is open to candidates in the U.S. or Canada. The engineer will own the architecture of core AI systems end-to-end, including service layers, data pipelines, and evaluation infrastructure. They will develop and maintain clinician-reviewed evaluation datasets, establish severity-weighted error metrics, build production monitoring and actionable alerts, and create adversarial tests. The role also collaborates with product and clinical leaders to set accuracy thresholds and determine when AI capabilities should be expanded or pulled back. The candidate is expected to actively adopt emerging AI tools to improve workflows and efficiency.

Requirements

  • At least 7 years of experience in software or machine learning engineering, including at least 3 years focused on evaluation, testing, or quality for AI systems.
  • Track record of designing and building complete systems from scratch and taking them from prototype to production.
  • Track record of building and running evaluation systems used by teams in production, rather than only for one-off analyses.
  • At least 2 years of experience evaluating large language model (LLM) systems in a production environment.
  • Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.
  • At least 1 year of experience designing evaluation sets for document extraction, classification, or structured output.
  • Deep knowledge of explainable AI (XAI) techniques and AI-powered tools such as SHAP, LIME, and Claude.
  • Clear written communication skills, including writing design documents, and ability to explain complex trade-offs to clinicians, engineers, and executives.
  • Curiosity and willingness to actively use a range of AI-enabled tools, such as copilots, chat-based AI, and automation solutions.

Nice To Haves

  • Experience working within healthcare or another highly regulated industry.
  • Familiarity with AI-specific testing techniques such as metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.
  • History of mentoring engineers and influencing technical direction across multiple teams.

Responsibilities

  • Build the core AI system end-to-end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.
  • Design and build evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.
  • Build and maintain evaluation datasets with clinician-reviewed correct answers, ensuring they evolve as systems change.
  • Create error metrics weighted by severity so that mistakes with clinical or financial impact are prioritized accordingly.
  • Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.
  • Write adversarial tests for AI systems, including tests for hidden or malicious instructions in documents and invented content.
  • Work with product and clinical leads to define accuracy thresholds and criteria for expanding or pulling back an AI capability.
  • Actively adopt and leverage emerging AI tools to improve workflows, solve problems, and drive efficiency.

Benefits

  • Comprehensive medical, dental, and vision benefits.
  • Three weeks of vacation plus five personal days.
  • Employee stock ownership.
  • RRSP program.
  • 401(k) matching.
  • Community involvement opportunities.
  • Flexible work arrangements.
  • Opportunities to grow professionally and learn.
  • Casual work environment.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service