ML Ops Engineer

Alaffia Health•New York, NY

3d•Hybrid

About The Position

Alaffia Health is a healthcare AI startup revolutionizing medical billing and data automation. Our AI-driven platform leverages state-of-the-art generative AI and machine learning technologies to enhance accuracy, efficiency, and compliance in medical billing and auditing. As we scale, we're looking for an MLOps Engineer to build the operational backbone behind Alaffia's AI agents. Our agentic systems are already in production reviewing real claims but as both our library of agents and our roster of health-plan customers grow, we need someone to make that growth reliable, measurable, and cost-efficient. This is a hybrid role. The core is MLOps: standing up evaluation frameworks, wiring evals into CI, managing model access across cloud providers, tracking experiments, and defining how we operate a growing fleet of agents across many customers. But we also want someone who can go deeper on the models themselves; developing, fine-tuning, and improving the LLMs and ML models that power our agents. If you like owning systems end to end from a model experiment in a dev environment all the way to a monitored, evaluated agent running in production, and want a direct hand in shaping how a Series B healthtech builds, ships, and measures AI, this role is for you. You'll work closely with our AI engineers and clinical subject-matter experts.

Requirements

5+ years of experience in MLOps, ML platforms, or ML infrastructure engineering, ideally with experience supporting LLM-based or agentic systems.
Strong software engineering fundamentals and proficiency in Python.
Hands-on experience developing, training, and fine-tuning machine learning models and/or LLMs.
Experience building evaluation or benchmarking frameworks for ML or LLM outputs.
Experience with CI/CD pipelines and integrating automated testing into them.
Working knowledge of cloud AI platforms such as AWS (Bedrock, SageMaker) and/or Azure (Azure OpenAI, Azure ML).
Familiarity with experiment-tracking and ML observability tooling (e.g., MLflow, Weights & Biases).
A solid grasp of the operational challenges specific to LLMs — non-determinism, latency, token cost, prompt and model versioning, and why evaluating them is genuinely hard.
A bias toward ownership and the ability to carry a system from experiment to production.

Nice To Haves

Experience operating multi-tenant production systems or ML services at scale.
Experience in a regulated or healthcare environment, including handling PHI and HIPAA considerations.
Experience with agent frameworks or orchestration.
Comfort collaborating with non-engineering domain experts, such as clinicians, to translate their expertise into model behavior.

Responsibilities

Design, build, and own our agent evaluation framework covering output quality, throughput, and cost, so we can measure every agent objectively and catch regressions early.
Integrate automated evals into our CI/CD pipeline so changes to our agents are tested and quality-gated before they ship.
Manage model access and enablement across cloud providers (AWS Bedrock, Azure OpenAI, and others), including onboarding new models, managing quotas, and benchmarking options for performance and reliability.
Build experiment tracking, metrics, and observability for the experiments our team runs in dev, so results are reproducible and comparable.
Develop the systems and processes to manage our agents across a growing roster of customers including versioning, configuration, and deployment as both the number of agents and customers scale.
Monitor agents in production for quality, throughput, cost, and drift, and build the dashboards and alerting to keep them healthy.
Develop, fine-tune, and evaluate LLMs and machine learning models to improve the accuracy and efficiency of our claim-review agents.
Build and maintain versioned pipelines for models, prompts, and agent configurations.
Track and optimize inference cost and unit economics as we scale.
Partner with AI engineers and clinical experts to turn domain expertise into measurable, deployable model behavior.