About The Position

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain systems that measure and improve the quality of its AI-powered applications. As AI capabilities expand across patient experiences, merchant tools, clinical workflows, and internal operations, this role will establish the infrastructure, evaluation methods, and monitoring practices needed to deploy AI features reliably. The role combines machine learning engineering, software infrastructure, automated testing, and production reliability. The engineer will build evaluation pipelines, benchmark datasets, regression tests, monitoring systems, and tools that help teams identify failures before they affect users. Evaluation will address accuracy, relevance, grounding, consistency, application requirements, and safety—not simply whether a model responds. The engineer will investigate production issues and collaborate with applied AI engineers to improve prompts, retrieval systems, model configurations, and application behavior. In healthcare-related workflows, the role will help establish safeguards, support human review where appropriate, and ensure AI systems meet defined quality expectations before deployment. This is a hands-on senior engineering position with ownership of evaluation infrastructure and reliability practices. The engineer will contribute production code, design scalable testing systems, establish reusable standards, mentor other engineers, and own improvements from technical design through deployment and ongoing maintenance.

Requirements

  • 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.
  • Demonstrated experience developing and operating machine learning or AI systems in production.
  • Strong Python programming skills and experience building production software services.
  • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
  • Strong understanding of machine learning evaluation methodologies and performance measurement.
  • Experience developing reproducible experiments and benchmark datasets.
  • Familiarity with large language models and modern LLM application architectures.
  • Understanding of prompt engineering, model inference, and AI application integration patterns.
  • Experience evaluating model outputs for quality, accuracy, and reliability.
  • Familiarity with retrieval-augmented generation and retrieval evaluation methods.
  • Experience with structured datasets, data pipelines, and analytical workflows.
  • Strong understanding of software testing, regression testing, and continuous integration.
  • Experience building or maintaining backend services and APIs.
  • Familiarity with cloud infrastructure and production deployment environments.
  • Experience with monitoring, logging, tracing, and production troubleshooting.
  • Understanding of model latency, throughput, and operational performance metrics.
  • Ability to investigate complex system failures and identify root causes.
  • Experience designing metrics and reporting systems for technical performance.
  • Familiarity with version control, code review, and collaborative software development.
  • Understanding of data quality and its impact on model evaluation.
  • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost.
  • Strong analytical skills and attention to detail.
  • Experience collaborating with engineers, product managers, and technical stakeholders.
  • Ability to independently own complex engineering initiatives.
  • Strong communication skills and ability to explain evaluation results clearly.

Nice To Haves

  • Experience building evaluation frameworks specifically for LLM-powered applications.
  • Familiarity with LangSmith, DeepEval, Ragas, or comparable evaluation platforms.
  • Experience implementing LLM-as-a-judge evaluation methods and validating their limitations.
  • Experience building automated hallucination detection or groundedness evaluation systems.
  • Familiarity with agent evaluation and multi-step workflow testing.
  • Experience evaluating tool-calling accuracy and structured model outputs.
  • Experience with retrieval metrics such as precision, recall, and ranking quality.
  • Familiarity with vector databases, embeddings, and semantic search.
  • Experience with MLflow, Weights & Biases, or similar experiment tracking tools.
  • Experience developing machine learning observability infrastructure.
  • Familiarity with model drift detection and data quality monitoring.
  • Experience integrating evaluation checks into CI/CD pipelines.
  • Experience with distributed processing or large-scale evaluation workloads.
  • Familiarity with AWS and cloud-based machine learning infrastructure.
  • Experience building internal developer tools for AI engineering teams.
  • Experience evaluating systems built with OpenAI, Anthropic, or comparable model providers.
  • Familiarity with healthcare, telehealth, or other sensitive-data applications.
  • Understanding of HIPAA and healthcare data privacy considerations.
  • Experience designing human-in-the-loop evaluation or review systems.
  • Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

  • Design and build evaluation infrastructure for AI and machine learning applications.
  • Develop automated testing systems for large language model applications and intelligent workflows.
  • Establish measurable quality standards for production AI features.
  • Create evaluation pipelines for accuracy, relevance, completeness, consistency, and reliability.
  • Build and maintain representative datasets for testing AI application behavior.
  • Develop regression testing systems for changes to models, prompts, and application logic.
  • Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems.
  • Measure retrieval quality, grounding accuracy, and unsupported model outputs.
  • Build tools to identify hallucinations, incorrect reasoning, and unreliable responses.
  • Develop testing approaches for AI agents, tool calling, and multi-step workflows.
  • Evaluate whether AI systems follow application instructions and respect workflow boundaries.
  • Implement automated checks for structured outputs, API interactions, and tool execution.
  • Build production monitoring systems and dashboards for AI quality, latency, errors, and operational reliability.
  • Investigate production AI failures and identify their technical causes.
  • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations.
  • Establish testing procedures for model upgrades and provider changes.
  • Develop benchmarking tools to compare models by quality, performance, and cost.
  • Implement evaluation processes for sensitive healthcare-related AI workflows.
  • Work with product and clinical stakeholders to define acceptance criteria and escalation requirements.
  • Support human review and feedback collection for AI outputs requiring additional validation.
  • Design reproducible experiments to evaluate AI system improvements.
  • Build reusable evaluation libraries, services, and internal engineering tools.
  • Integrate AI evaluation checks into development and deployment workflows.
  • Improve observability through structured logging, tracing, and production diagnostics.
  • Mentor engineers on AI testing methodologies and reliability practices.
  • Contribute to architecture discussions about AI infrastructure, testing, and monitoring.
  • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service