ML Platform Engineer, Evaluation Systems

Bask Health

About The Position

Bask Health is seeking an ML Platform Engineer to build and maintain infrastructure for reliable, consistent, and scalable evaluation of machine learning models and AI-powered applications. The role focuses on platform services, data pipelines, execution environments, and monitoring that enable engineering teams to measure model quality, compare changes, and assess AI systems before and after deployment. Responsibilities may include developing evaluation job orchestration, managing benchmark datasets, building experiment tracking systems, supporting model comparisons, and creating infrastructure to store and analyze results. The engineer will work with cloud services, distributed processing, model integrations, databases, and automated workflows to enable repeatable experiments and large-scale evaluation jobs. The role collaborates with machine learning engineers, applied scientists, AI evaluation engineers, and backend engineers to build reusable systems for different model types and evaluation methodologies. It emphasizes reproducibility, versioned datasets and configurations, observable failures, accurate comparisons over time, and secure handling of sensitive information in healthcare-related environments. The engineer will contribute to architecture and own platform projects through design, deployment, monitoring, and continuous improvement.

Requirements

At least 3 years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related technical field.
Experience building and maintaining production backend services or infrastructure.
Strong programming skills in Python, TypeScript, or comparable languages.
Familiarity with machine learning development and evaluation workflows.
Experience developing APIs and backend service integrations.
Understanding of distributed systems and asynchronous processing.
Experience with cloud-based infrastructure and production deployments.
Familiarity with automated testing and continuous integration practices.
Understanding of machine learning model versioning and experiment reproducibility.
Experience with relational databases and structured datasets.
Familiarity with data pipelines and batch processing workflows.
Experience designing reliable background job systems.
Understanding of infrastructure scalability, performance, and fault tolerance.
Familiarity with logging, monitoring, and production observability.
Experience implementing retries, timeouts, and error recovery.
Understanding of authentication, authorization, and secure service communication.
Familiarity with configuration management and deployment automation.
Experience writing maintainable, well-tested production code.
Ability to troubleshoot complex backend or infrastructure issues.
Understanding of software release processes and operational reliability.
Experience using Git and collaborative engineering workflows.
Ability to translate machine learning requirements into scalable infrastructure.
Strong analytical thinking and problem-solving skills.
Ability to collaborate effectively with machine learning engineers, applied scientists, and software engineers.
Ability to independently own engineering projects from design through deployment.

Nice To Haves

Experience building infrastructure for machine learning evaluation or experimentation.
Familiarity with MLflow, Weights & Biases, or comparable experiment tracking platforms.
Experience with model registries and artifact management systems.
Familiarity with large language model evaluation frameworks.
Experience with LangSmith, DeepEval, Ragas, or comparable AI evaluation tools.
Experience building distributed evaluation or benchmarking pipelines.
Familiarity with workflow orchestration tools such as Airflow, Dagster, or Temporal.
Experience with containerized workloads and distributed compute infrastructure.
Familiarity with Docker, Kubernetes, or comparable technologies.
Experience with AWS Lambda, SQS, EventBridge, or related cloud services.
Familiarity with AWS SageMaker or comparable ML platform services.
Experience with serverless infrastructure and infrastructure-as-code tools.
Familiarity with SST or comparable application deployment frameworks.
Experience developing data versioning and dataset management systems.
Understanding of reproducible machine learning experimentation.
Experience implementing model performance monitoring and drift detection.
Familiarity with large language model APIs and model comparison workflows.
Experience building internal developer platforms for machine learning teams.
Understanding of HIPAA and healthcare data privacy considerations.
Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

Design, build, and maintain machine learning evaluation infrastructure.
Develop platform services for executing and managing automated model evaluation workloads.
Build scalable systems for large batches of model and AI application evaluations.
Implement job orchestration, scheduling, and execution management for evaluation pipelines.
Develop infrastructure for storing and versioning evaluation datasets.
Track model configurations, prompts, parameters, and evaluation results.
Implement reproducible experiment workflows for consistent model comparisons.
Develop automated model benchmarking pipelines.
Build APIs and services for submitting and monitoring evaluation jobs.
Create systems for comparing evaluation results across model versions and experiments.
Support evaluations of large language models, predictive models, and retrieval systems.
Develop asynchronous processing systems for computationally intensive evaluation tasks.
Monitor evaluation job performance, failures, latency, and resource usage.
Build logging and tracing capabilities to investigate evaluation problems.
Manage evaluation artifacts, reports, and historical results.
Integrate evaluation infrastructure with model registries and ML development workflows.
Implement automated quality checks for controlled model releases.
Build retry, timeout, and failure-recovery mechanisms for evaluation workloads.
Optimize evaluation performance, infrastructure utilization, and operational costs.
Support distributed evaluation workloads across cloud compute resources.
Collaborate with machine learning engineers to define infrastructure requirements and with applied scientists on reproducible experiments and benchmark comparisons.
Develop tools that simplify evaluation configuration and execution.
Implement secure access controls for evaluation datasets and results.
Ensure evaluation systems respect tenant boundaries and approved data-handling practices.
Investigate infrastructure incidents and improve platform reliability.
Document infrastructure, deployment procedures, and operational standards.
Own assigned ML evaluation platform projects from technical design through deployment, monitoring, and continuous improvement.