Staff Machine Learning Engineer

Plenful • San Francisco, CA

1d • Hybrid

About The Position

Plenful is building an AI automation platform for healthcare operations, helping hospitals, health systems, pharmacies, and payors reduce manual work, administrative burden, and compliance challenges. The Staff Machine Learning Engineer will set the technical direction for building, training, and measuring production AI across the company while remaining hands-on. This is an individual-contributor role with no direct reports and broad organizational influence.

The role leads the architecture of production agents and the company’s post-training and evaluation strategies. It also establishes systems that turn production failures and user feedback into improved datasets, evaluations, and models; sets standards for reproducibility, observability, and MLOps; and co-owns the ML roadmap with engineering and product leaders. The engineer will lead technical reviews, mentor senior engineers, contribute to hiring, and build privacy-conscious practices suited to healthcare data.

The first 30 days include reviewing agents, evaluations, and the training pipeline and documenting the three largest gaps. By day 90, the engineer is expected to have shipped a post-training or evaluation improvement that moves a production metric and published the team’s plan for the following two quarters. The R&D role follows a hybrid model, with two days per week in the San Francisco office; Plenful describes itself as remote-first, with office presence in San Francisco and New York.

Requirements

• At least 8 years of experience in ML or software engineering, including at least 3 years leading LLM or agentic systems in production.
• Deep applied post-training experience, including end-to-end supervised fine-tuning, preference optimization, distillation, or reinforcement learning, with a shipped model that improved a production metric.
• Experience designing an evaluation strategy for LLM or agent systems and using it to drive real release decisions.
• Experience architecting production agent systems and understanding their failure modes and remediation.
• Experience with frameworks such as LangGraph, retrieval-augmented generation (RAG), and vector search; Pinecone is given as an example.
• Strong Python and PyTorch skills.
• Comfort with distributed training and serving, with vLLM, FSDP, and DeepSpeed given as examples.
• Strong MLOps experience, including Docker, Kubernetes, MLflow or an equivalent, CI/CD, and experiment and dataset tracking.
• Ability to influence without authority through design documents, reviews, and mentoring.
• Ability to stay current with new models and coding agents, quickly test new research and releases, and report results with numbers.
• Evidence of technical contribution, such as open-source work, write-ups, talks, or evaluations used by others.

Nice To Haves

• Healthcare data experience, such as EHR, FHIR, claims, or prior authorization.
• Publications or open-source work in LLM evaluation, post-training, or agents.
• Experience interviewing and hiring ML engineers.
• Experience in another regulated, high-stakes domain.

Responsibilities

• Set the architecture and reusable patterns for production agents, including tool use, retrieval, state and context management, routing, orchestration, tracing, and failure recovery.
• Make build-versus-buy decisions for models and platform components.
• Own post-training strategy, deciding when supervised fine-tuning, preference optimization, distillation, and reinforcement learning are appropriate.
• Train and ship LLMs and vision-language models for healthcare tasks, building internal capability to continue this work.
• Demonstrate model improvements through production-relevant evaluation.
• Set AI quality strategy using automated graders, calibrated LLM judges, regression suites, human review, failure taxonomies, and offline-to-online validation; use evaluation results to inform release decisions.
• Design feedback and data systems that turn production failures and user feedback into better datasets, evaluations, and models.
• Set standards for dataset versioning, observability, and MLOps to make work reproducible and measurable.
• Co-own the 12-month ML roadmap with engineering and product leaders.
• Lead technical reviews, mentor senior engineers, and help hire.
• Build privacy into systems through minimum-necessary data use, de-identified or synthetic data where appropriate, and HIPAA-aware practices.
• Review the company’s agents, evaluations, and training pipeline, and document the three biggest gaps by day 30.
• Ship a post-training or evaluation improvement that moves a production metric and publish the team’s plan for the following two quarters by day 90.

Benefits

• Full medical, dental, and vision insurance, with family participation.
• 401(k) with a company match of 50% of the first 3% contributed.
• Equity for every full-time employee.
• Unlimited PTO.
• Daily lunch stipend of $100 per week.
• Wellness stipend of $100 per month.
• Commuter benefits of $100 per month for employees based in San Francisco and New York.
• Paid parental leave.
• Flexible hybrid work environment; R&D roles work two days per week in the San Francisco office.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed