About The Position

Bask Health is seeking a hands-on Machine Learning Engineer, Evaluation & Reliability to help ensure that AI-powered features are accurate, dependable, and suitable for real-world use. The role focuses on developing evaluation infrastructure, testing workflows, and performance measurements for large language models, retrieval systems, AI assistants, and automated workflows. Work includes creating evaluation datasets, implementing benchmarks, measuring response quality, testing retrieval accuracy, analyzing failures, and monitoring production behavior. The engineer will collaborate with applied AI, machine learning, and backend engineers, product managers, and other stakeholders to define successful AI behavior and translate expectations into measurable tests and engineering improvements. Evaluation must account for accuracy, privacy, permissions, and the risks of unreliable responses in healthcare-related environments, including unsupported information, restricted-data exposure, unauthorized actions, and behavior outside approved boundaries. This position emphasizes practical implementation and continuous improvement. The engineer will contribute to evaluation methodologies and engineering standards, work with senior engineers on reliability strategies and complex technical decisions, and own assigned projects from implementation through deployment and ongoing improvement.

Requirements

  • At least 3 years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.
  • Experience building and maintaining production software or machine learning applications.
  • Strong programming skills in Python, TypeScript, or comparable languages.
  • Familiarity with large language models and modern generative AI applications.
  • Experience testing or evaluating machine learning models or AI-powered software.
  • Understanding of machine learning evaluation metrics and experimental methodologies.
  • Experience developing automated tests and validation pipelines.
  • Familiarity with prompt engineering and model output analysis.
  • Ability to analyze structured and unstructured datasets.
  • Experience writing scripts or services for data processing and evaluation.
  • Understanding of model accuracy, precision, recall, and other applicable performance measures.
  • Familiarity with API integrations and backend application development.
  • Experience working with relational databases or analytical data systems.
  • Ability to identify patterns in model failures and translate findings into engineering improvements.
  • Understanding of regression testing and software quality assurance.
  • Familiarity with logging, monitoring, and production troubleshooting.
  • Experience using Git and collaborative software development workflows.
  • Understanding of the limitations and failure modes of large language models.
  • Ability to design representative test cases and evaluation scenarios.
  • Familiarity with cloud-based development and deployment environments.
  • Experience writing maintainable, well-tested production code.
  • Strong analytical thinking and attention to detail.
  • Ability to communicate technical evaluation results clearly.
  • Experience collaborating with engineers, product managers, and cross-functional teams.
  • Ability to independently own assigned technical projects.

Nice To Haves

  • Experience building evaluation frameworks for large language models.
  • Familiarity with OpenAI, Anthropic, or comparable model APIs.
  • Experience with LangSmith, DeepEval, Ragas, or comparable AI evaluation frameworks.
  • Familiarity with LLM-as-a-judge evaluation techniques.
  • Experience creating human-labeled evaluation datasets.
  • Understanding of inter-rater agreement and evaluation consistency.
  • Experience evaluating retrieval-augmented generation systems.
  • Familiarity with embeddings, semantic search, and vector databases.
  • Experience evaluating AI agents and tool-calling workflows.
  • Familiarity with prompt injection testing and AI security evaluations.
  • Experience implementing automated regression testing for AI applications.
  • Familiarity with experiment tracking and model comparison tools.
  • Experience building dashboards for machine learning performance monitoring.
  • Understanding of statistical analysis and experiment design.
  • Experience working with Python data analysis libraries.
  • Familiarity with TypeScript and Node.js application development.
  • Experience working with AWS or serverless infrastructure.
  • Experience evaluating AI systems used in healthcare or other sensitive environments.
  • Understanding of HIPAA and healthcare data privacy considerations.
  • Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

  • Build and maintain evaluation systems for AI-powered applications.
  • Develop automated testing frameworks for large language models and AI application workflows.
  • Create and maintain representative datasets for evaluating model behavior.
  • Implement benchmarks for response accuracy, relevance, completeness, and consistency.
  • Develop evaluation pipelines for prompts, retrieval systems, and model configurations.
  • Test AI assistants and agents against realistic user scenarios and workflow requirements.
  • Build automated regression tests to detect performance changes between releases.
  • Measure hallucination rates, unsupported claims, and other model quality failures.
  • Evaluate retrieval-augmented generation systems for document relevance and answer grounding.
  • Develop tests for structured model outputs and schema compliance.
  • Evaluate tool-calling accuracy and agent action selection.
  • Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
  • Implement evaluation checks for sensitive patient and operational information.
  • Build dashboards and reports that communicate AI quality metrics to engineering teams.
  • Analyze evaluation results and identify recurring failure patterns.
  • Investigate production incidents involving inaccurate or unreliable AI outputs.
  • Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.
  • Support model comparisons and controlled experiments.
  • Evaluate tradeoffs between model quality, latency, reliability, and inference cost.
  • Develop automated quality checks for CI/CD and AI feature release workflows.
  • Implement monitoring and alerting for production AI performance.
  • Build tools for reviewing, labeling, and analyzing model responses.
  • Maintain evaluation datasets as products and user behavior evolve.
  • Document evaluation methodologies, test cases, and known model limitations.
  • Work with product teams to define measurable acceptance criteria for AI features.
  • Collaborate with senior engineers on evaluation standards and reliability improvements.
  • Participate in code reviews, technical discussions, and engineering planning.
  • Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service