Customer Support Specialists, AI Response Evaluation

mpathic•Seattle, WA

11d•$30 - $40

About The Position

mpathic is seeking part-time Customer Support Specialists to support a red-teaming and quality assurance (QA) campaign focused on evaluating AI system behavior in customer-facing interactions. In this role, you will review AI-generated responses and multi-turn conversations to assess quality, accuracy, tone, and safety—similar to evaluating customer support agents. You will identify issues such as incorrect information, poor customer experience, inappropriate tone, or failure to resolve user needs effectively. This is not a traditional support role. Instead, it focuses on evaluating AI performance, applying your customer support expertise to assess how well AI systems handle real-world customer interactions. You will help identify, prevent, and characterize risks that emerge when users interact with AI-powered support systems. This role requires strong judgment, attention to detail, and the ability to evaluate nuanced customer interactions.

Requirements

Professional experience in one or more of the following: Customer support (chat, email, or phone) Call centers or contact centers Technical support or help desk roles Online support environments (SaaS, e-commerce, fintech, etc.)
Strong understanding of: High-quality customer experience and service standards Effective communication in customer-facing interactions Common support workflows (triaging, troubleshooting, escalation) How customers interpret and respond to support interactions
Ability to identify: Poor or incomplete resolutions Incorrect or misleading information Inappropriate tone (e.g., robotic, dismissive, overly casual) Lack of empathy or ineffective de-escalation Failure to follow support best practices
Strong written communication skills and ability to clearly explain reasoning
Experience with or interest in: Evaluating conversations or QA for support teams Working with AI tools, chatbots, or automated systems Assessing how systems perform under real customer behavior
Comfort with: Tech tools and platforms (Slack, spreadsheets, dashboards) Evaluating AI-generated responses (no coding required, but must be tech-comfortable) Ambiguity, iteration, and feedback-driven workflows
Willingness to: Sign NDAs and work with sensitive content

Nice To Haves

Experience in QA, coaching, or training within support teams
Background in trust & safety or content moderation
Experience with chatbot or AI-assisted support tools
Familiarity with CX metrics (CSAT, QA scoring, resolution rate)
Interest in AI, automation, or improving customer experience systems

Responsibilities

Reviewing AI-generated customer support responses for accuracy, clarity, and helpfulness
Evaluating tone, empathy, and professionalism in customer interactions
Identifying poor support behaviors (e.g., dismissiveness, lack of resolution, incorrect guidance)
Assessing whether AI responses appropriately address customer intent and resolve issues
Evaluating how AI handles frustration, confusion, or escalating user behavior
Identifying gaps in responses, including missing steps, unclear instructions, or lack of follow-through
Detecting overconfident, misleading, or incorrect responses
Evaluating multi-turn conversations for consistency, escalation handling, and resolution quality
Identifying patterns of agreement without problem-solving (e.g., empty empathy or unhelpful reassurance)
Participating in or reviewing red teaming exercises to surface failure modes in support scenarios
Supporting quality assurance (QA) to ensure consistency across evaluations
Documenting edge cases, failure patterns, and customer experience risks
Providing structured written feedback to internal teams
Collaborating with interdisciplinary teams on AI quality, safety, and evaluation frameworks
Maintaining strict confidentiality and quality standards