AI Agent Experience Designer

Allied Benefit Systems•Chicago, IL

1d•Remote

About The Position

The Prompt Engineer will be responsible for optimizing, evaluating, and advancing the performance of AI-driven agents, such as chatbots and autonomous workflows to ensure they deliver accurate, efficient, and high-quality outcomes. This role combines analytical thinking, user experience insight, and technical expertise to monitor agent behavior, identify improvement opportunities, and implement enhancements that align with business goals.

Requirements

Bachelor’s degree in Computer Science, Math, Statistics, UI/UX, Process Design or equivalent work experience required.
6+ years of experience across business analysis, conversational AI, UX writing, and related product roles.
Direct experience designing and improving chatbots, voicebots, IVR modernization, or contact center automation.
Ability to translate business workflows into clear user requests, key information to capture (e.g., member identifiers, dates, plan options), and end-to-end conversation paths that engineering can build and QA can test.
Strong prompt design skills and a test-and-learn mindset using real interaction data.
Strong writing and editing skills with comfort in a regulated, customer-facing environment.
Comfort operating with ambiguity in a new and evolving role, and able to create processes and standards where none exist.
Ability to collaborate with cross-functional stakeholders and external partners.
Shipped conversational AI experiences in voice and/or contact center environments.
Experience working with plan documents, knowledge bases, and policy text in regulated environments.
Able to translate source material into grounded answers and guardrails.

Nice To Haves

Requirements and functional specifications for prioritized conversational AI use cases.
Use case catalog and key information requirements mapped to approved scenarios.
Conversation flow designs that include fallbacks, error handling, and escalation.
Versioned prompt library with testing notes and a clear change history.
UAT scripts and acceptance criteria, plus defect tracking through resolution.

Responsibilities

Define and document required AI agent behavior, including edge cases, compliance constraints, escalation expectations, and response quality targets.
Convert benefits workflows and policy rules into implementable use cases, key data fields to capture, and testable scenarios that can be built and verified.
Maintain a clear mapping from requests to delivered behavior so gaps are visible and reviewable.
Drive stakeholder reviews and acceptance of behavior, flows, and decision logic before changes ship.
Craft and iterate prompt instructions and response rules that produce accurate, consistent answers in a regulated benefits context.
Build reusable prompt patterns and maintain a controlled prompt library with clear version history.
Review real interactions to identify failure modes, then adjust prompts, context, and fallbacks to improve outcomes.
Partner with engineering and QA to define regression test coverage and release criteria for prompt and flow changes.
Define the agent’s voice and interaction style so it is professional, on-brand, and easy for members and providers to follow.
Design multi-step conversation paths that handle ambiguity, corrections, and interruptions without losing the user.
Define when the agent should answer, when it should ask clarifying questions, and when it should hand off to a live person.
Validate that wording, terminology, and risk posture match benefits operations expectations.
Define how the agent should use plan documents to answer questions, including what to do when source material is missing or ambiguous.
Partner with contact center and workforce management teams to align the agent experience with SOPs and ensure workflows stay current as procedures change.
Partner with engineering on query shaping and grounding so document-based answers stay tied to supporting text.
Own the gold standard definition of intended agent behavior after launch, so teams can support and improve it without relying on vendor or institutional knowledge.
Maintain the core operating artifacts used day to day: use case catalog, flow maps, escalation rules, prompt library, and a running list of known failure modes.
Create lightweight training and working sessions for internal support partners and business SMEs.
Establish a change process for updating prompts and flows, including testing expectations, approvals, and version history.
Maintain a curated set of golden records and test cases that represent critical scenarios and edge cases and use them to support automated regression and quality testing.
Operator documentation that enables internal support and ongoing improvement.
Post-launch performance monitoring, including top failure modes and improvement plans.
Other duties as assigned