Senior Manager, AI Performance & Operations
About The Position
As a Senior Manager, AI Performance & Operations, you will own the production performance and continuous improvement of Devoted’s core agentic workflows for service and experience. so that every general-availability workflow has clear performance thresholds, reliable monitoring, explicit fallback paths, and a measurable improvement plan. This person is the accountable operator for Guide’s highest-volume and highest-risk AI workflows once they are live. They maintain the portfolio view of where each workflow stands, detect degradation early, diagnose the root cause, drive corrective action, and verify that implemented changes improve outcomes. They also serve as the in-house expert for early-stage projects by helping teams define instrumentation, evaluation design, launch criteria, and safe hand off into general availability.
Requirements
- Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems.
- Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows.
- A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic.
- Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage.
- Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments.
- Deep personal accountability with a drive to own results rather than just producing recommendations.
Nice To Haves
- Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions.
Responsibilities
- Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met.
- Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions.
- Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently.
- Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure.
- Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates.
- Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changes.
- Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production.
- Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models.
- Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it.
Benefits
- Employer sponsored health, dental and vision plan with low or no premium
- Generous paid time off
- $100 monthly mobile or internet stipend
- Stock options for all employees
- Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
- Parental leave program
- 401K program
- And more....
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
