Staff Machine Learning Engineer

Plenful•San Francisco, CA

1d•Hybrid

About The Position

Plenful is hiring a Staff Machine Learning Engineer to set technical direction for building, training, and evaluating AI across the company while remaining hands-on. This is an individual-contributor role with no direct reports and broad influence. The engineer will guide production agent architecture, own the post-training strategy for LLMs and vision-language models, define AI quality and evaluation standards, and build systems that turn production failures and user feedback into improved datasets, evaluations, and models. The role also helps shape the ML roadmap, mentors senior engineers, supports hiring, and establishes privacy-aware practices for healthcare data. During the first 30 days, the engineer is expected to review the agents, evaluations, and training pipeline and identify the three biggest gaps. By day 90, the engineer is expected to ship a post-training or evaluation improvement that moves a production metric and publish the plan the team will follow for the next two quarters. Plenful is remote-first, with meaningful office presence in San Francisco and New York. R&D roles follow a hybrid model requiring two days per week in the San Francisco office.

Requirements

At least 8 years of experience in ML or software engineering, including at least 3 years leading LLM or agentic systems in production.
End-to-end applied post-training experience, including supervised fine-tuning, preference optimization, distillation, or reinforcement learning, with a shipped model that improved a production metric.
Experience designing an evaluation strategy for LLM or agent systems and using it to drive real release decisions.
Experience architecting production agent systems and diagnosing and addressing their failures.
Expected experience with agent frameworks such as LangGraph, RAG, and vector search such as Pinecone.
Strong Python and PyTorch skills.
Comfort with distributed training and serving, such as vLLM, FSDP, or DeepSpeed.
Strong MLOps experience, including Docker, Kubernetes, MLflow or equivalent, CI/CD, and experiment and dataset tracking.
Ability to influence without authority through design documents, reviews, and mentoring.
Ability to stay current with new models and coding agents and quickly turn papers and releases into experiments with measurable results.
Evidence of relevant work through open-source contributions, write-ups, talks, or evaluations used by others.

Nice To Haves

Healthcare data experience, including EHR, FHIR, claims, or prior authorization.
Publications or open-source work in LLM evaluation, post-training, or agents.
Experience interviewing and hiring ML engineers.
Experience working in another regulated, high-stakes domain.

Responsibilities

Define production-agent architecture, including tool use, retrieval, state and context management, routing, orchestration, tracing, and failure recovery.
Develop reusable engineering patterns and make build-versus-buy decisions for models and platform components.
Set post-training strategy, deciding when to use supervised fine-tuning, preference optimization, distillation, or reinforcement learning.
Train and ship LLMs and vision-language models for healthcare tasks and build internal capability to continue doing so.
Validate model improvements with evaluations that hold up in production.
Set AI quality strategy using automated graders, calibrated LLM judges, regression suites, human review, failure taxonomies, and offline-to-online validation.
Use evaluation results as the basis for release decisions.
Design feedback loops that turn production failures and user feedback into better datasets, evaluations, and models.
Set standards for dataset versioning, observability, and MLOps to make work reproducible and measurable.
Co-own the 12-month ML roadmap with engineering and product leaders.
Lead technical reviews, mentor senior engineers, and help with hiring.
Build privacy into systems through minimum-necessary data, de-identified or synthetic data where appropriate, and HIPAA-aware practices.
Review existing agents, evaluations, and training pipeline; identify major gaps; and deliver a production-metric improvement and two-quarter plan within the first 90 days.
Experiment with new models and coding agents, translate papers and releases into experiments, and share results with colleagues.

Benefits

Full medical, dental, and vision insurance, with family participation.
401(k) with company match of 50% of the first 3% contributed.
Equity for every full-time employee.
Unlimited PTO.
Daily lunch stipend of $100 per week.
Wellness stipend of $100 per month.
Commuter benefits of $100 per month for San Francisco- and New York-based employees.
Paid parental leave.
Growth opportunities through collaboration with experienced, high-performing leaders.
Flexible hybrid work environment; R&D roles work two days per week in the San Francisco office.