AI Evaluation Engineer

Bask Health

16h

About The Position

Bask Health is a software platform that helps entrepreneurs, doctors, and healthcare businesses launch, manage, and grow direct-to-consumer telehealth businesses. Its platform connects patient experiences, clinical workflows, pharmacy operations, and commerce tools. The engineering team is entrepreneurial, product-focused, AI-first, and builder-led, with engineers working closely with product teams and company leadership to take features from idea to production. The AI Evaluation Engineer will help ensure AI-powered features behave accurately, reliably, and consistently before and after production release. This hands-on role combines AI evaluation, software testing, automation, and production quality assurance. The engineer will build and maintain workflows, datasets, test scenarios, automated checks, and tools for evaluating large language models, AI assistants, retrieval systems, and other automated features against product requirements. The role includes evaluating generated responses, structured outputs, tool-calling behavior, retrieval accuracy, and regressions associated with prompt, model, or application-code changes. The engineer will collaborate with applied AI, machine learning, backend, product, and quality-focused engineering teams to define expected behavior, investigate failures, reproduce problems, and verify fixes. They will also contribute to processes for reviewing AI failures, documenting recurring issues, and assessing acceptance criteria. Because Bask operates in healthcare-related environments, the engineer will help verify privacy, permissions, and operational boundaries, including avoiding unsupported medical claims and interference with approved clinical or pharmacy workflows. The role emphasizes practical testing, thoughtful engineering, continuous improvement, and independent ownership of evaluation tools and tests.

Requirements

At least 3 years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
Experience developing automated tests for production software applications.
Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
Familiarity with large language models and modern AI-powered applications.
Understanding of software testing methodologies and test case design.
Experience building automated testing scripts, frameworks, or validation tools.
Familiarity with API testing and backend application integrations.
Understanding of prompt engineering and common AI model failure modes.
Ability to evaluate AI-generated responses against expected requirements.
Experience working with structured data and validating application outputs.
Familiarity with regression testing and software release processes.
Experience debugging application failures and reproducing technical issues.
Understanding of authentication, authorization, and secure application behavior.
Familiarity with databases and application data models.
Experience working with Git and collaborative software development workflows.
Understanding of continuous integration and automated testing pipelines.
Familiarity with logging, monitoring, and production troubleshooting.
Ability to create clear, reproducible test cases.
Strong analytical thinking and attention to detail.
Ability to distinguish application defects from model behavior limitations.
Experience documenting technical issues and communicating findings.
Familiarity with cloud-based application environments.
Ability to collaborate effectively with engineers, product managers, and other stakeholders.
Strong problem-solving skills and a quality-focused engineering mindset.
Ability to independently own evaluation and test automation projects.

Nice To Haves

Experience evaluating large language models or generative AI applications.
Familiarity with OpenAI, Anthropic, or comparable model APIs.
Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
Familiarity with LLM-as-a-judge evaluation techniques.
Experience developing AI evaluation datasets and test benchmarks.
Understanding of human evaluation and response labeling workflows.
Experience testing retrieval-augmented generation systems.
Familiarity with embeddings, semantic search, and vector databases.
Experience evaluating AI agents and tool-calling behavior.
Familiarity with prompt injection testing and AI security evaluations.
Experience implementing AI regression tests in CI/CD pipelines.
Experience building automated evaluation reports or dashboards.
Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.
Experience testing TypeScript or Node.js applications.
Familiarity with React or Next.js application testing.
Experience working with AWS or serverless infrastructure.
Experience evaluating AI features in healthcare or other sensitive environments.
Understanding of HIPAA and healthcare data privacy considerations.
Experience developing human-in-the-loop quality review processes.
Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

Design and implement evaluation workflows for AI-powered applications across Bask Health.
Build automated tests for large language model outputs and AI application behavior.
Develop evaluation datasets representing realistic user requests and product workflows.
Create test scenarios for patient-facing, merchant-facing, and internal AI features.
Implement checks for response accuracy, relevance, completeness, and consistency.
Evaluate AI-generated content for unsupported claims and misleading information.
Test structured model outputs for schema compliance and application compatibility.
Build evaluations for AI tool calling and authorized action execution.
Validate that AI assistants respect application permissions and tenant boundaries.
Test retrieval-augmented generation systems for grounding and information relevance.
Develop regression tests that identify unexpected changes in AI behavior.
Integrate evaluation checks into engineering development and release workflows.
Create automated comparison tests for prompts, model versions, and configuration changes.
Develop scripts and utilities to simplify AI evaluation execution.
Build reporting tools that summarize evaluation results and failure patterns.
Investigate failed evaluations and help engineers reproduce underlying issues.
Work with product managers to define expected AI behavior and acceptance criteria.
Collaborate with applied AI engineers to improve prompts and application logic.
Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
Develop evaluation cases for prompt injection and unauthorized tool-use attempts.
Support human review workflows for outputs requiring subjective assessment.
Track recurring AI failures and help prioritize reliability improvements.
Monitor production evaluation signals and investigate quality regressions.
Document test coverage, evaluation procedures, and known limitations.
Improve evaluation efficiency, repeatability, and maintainability.
Participate in code reviews, engineering planning, and technical discussions.
Support controlled testing and rollout of new AI-powered features.
Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.