Software Engineer, AI Quality & Evaluation

Bask Health

16h

About The Position

Bask Health is seeking a hands-on software engineer to build systems that help engineering teams test, evaluate, and improve AI-powered applications throughout development and production. The role focuses on reusable infrastructure—including backend services, APIs, testing frameworks, automation, dataset management, evaluation result storage and comparison, and tools for investigating AI failures—rather than evaluating individual models in isolation. The engineer will integrate AI evaluation into standard engineering workflows so teams can run tests during development, compare model or prompt changes, inspect failures, and identify regressions before release. The position works closely with applied AI engineers, machine learning engineers, AI evaluation specialists, and backend engineers to translate testing needs into dependable software capabilities. Because Bask works in healthcare-related environments, the role also supports appropriate handling of sensitive information, access permissions, privacy, safety, and approved operational boundaries. The engineer will contribute to technical architecture, maintainable code, infrastructure performance, developer experience, and production reliability, and will own assigned projects from design through deployment, monitoring, and continuous improvement.

Requirements

At least 3 years of professional software engineering experience, preferably in backend, platform, infrastructure, or developer tooling roles.
Experience building and maintaining production backend services.
Strong programming skills in TypeScript, JavaScript, Python, or comparable languages.
Experience designing and developing APIs and service integrations.
Familiarity with automated testing frameworks and software quality practices.
Understanding of continuous integration and deployment workflows.
Experience building reusable software components or internal engineering tools.
Familiarity with large language models and modern AI application development.
Understanding of the challenges involved in testing nondeterministic AI systems.
Experience with relational databases and structured application data.
Familiarity with asynchronous processing and background job systems.
Understanding of distributed systems fundamentals and service reliability.
Experience developing automated testing pipelines.
Familiarity with cloud-based application deployment.
Understanding of authentication, authorization, and secure service communication.
Experience implementing logging, monitoring, and production diagnostics.
Ability to troubleshoot backend services and infrastructure issues.
Familiarity with configuration management and software release processes.
Experience writing maintainable, well-tested production code.
Understanding of API error handling, retries, and failure recovery.
Experience working with Git and collaborative engineering workflows.
Ability to translate technical requirements into reusable engineering systems.
Strong analytical thinking and problem-solving skills.
Ability to collaborate effectively with software engineers, AI engineers, and machine learning specialists.
Ability to independently own engineering projects from technical design through deployment.

Nice To Haves

Experience building infrastructure for AI evaluation or machine learning testing.
Familiarity with OpenAI, Anthropic, or comparable model APIs.
Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
Familiarity with LLM-as-a-judge evaluation approaches.
Experience building evaluation execution services or automated benchmark systems.
Familiarity with AI regression testing and prompt comparison workflows.
Experience developing internal developer platforms or SDKs.
Experience building testing infrastructure for distributed applications.
Familiarity with TypeScript and Node.js backend development.
Experience with React or Next.js for internal tooling interfaces.
Familiarity with tRPC, Prisma, Drizzle, or comparable development tools.
Experience working with AWS Lambda, SQS, EventBridge, or related cloud services.
Familiarity with serverless infrastructure and infrastructure-as-code tools.
Experience with SST or comparable application deployment frameworks.
Experience building scalable asynchronous job processing systems.
Familiarity with embeddings, vector databases, and retrieval-augmented generation.
Experience implementing model observability or AI request tracing.
Understanding of multi-tenant SaaS architecture and data isolation.
Familiarity with HIPAA and healthcare data privacy considerations.
Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

Design, build, and maintain software infrastructure for AI quality and evaluation across Bask Health.
Develop backend services that execute and manage automated AI evaluations.
Build reusable testing tools for large language model applications.
Create APIs for submitting evaluation jobs and retrieving results.
Develop systems for managing evaluation datasets, test cases, and expected outcomes.
Build automated workflows for comparing prompts, models, and application configurations.
Implement AI regression testing infrastructure and integrate evaluation checks into CI/CD pipelines.
Develop services to record and organize evaluation results over time.
Build tools for engineers to inspect AI responses, failures, and test histories.
Implement structured result storage and comparison capabilities, including internal dashboards or interfaces for reviewing outcomes.
Create reusable software libraries that simplify AI testing across applications.
Build systems supporting deterministic checks and model-based evaluations.
Implement automated testing for tool-calling behavior and structured model outputs.
Develop infrastructure for evaluating retrieval-augmented generation applications.
Build asynchronous processing systems for large evaluation workloads, including scheduling, retries, timeouts, and error handling.
Develop monitoring and logging for evaluation infrastructure.
Optimize evaluation execution speed, reliability, and infrastructure costs.
Work with AI evaluation engineers to turn testing requirements into reusable software tools.
Collaborate with machine learning engineers on model and prompt comparison workflows.
Implement appropriate access controls and secure handling of evaluation data; ensure tenant boundaries and data privacy requirements are respected.
Investigate infrastructure failures and resolve issues affecting evaluation reliability.
Improve developer documentation and internal AI testing tools.
Participate in code reviews, technical planning, and architecture discussions.
Own assigned AI quality engineering projects from design through deployment, monitoring, and continuous improvement.