Red Teaming Expert

mpathic•Seattle, WA

12d•$30 - $60

About The Position

mpathic is seeking part-time, project-based Red Teaming Experts to support a red-teaming and evaluation campaign focused on AI safety and model behavior in sensitive, real-world interactions. In this role, you will design, simulate, and evaluate conversations with AI systems to assess safety, risk, and behavioral performance. You will identify failure modes, edge cases, and policy gaps—particularly in scenarios involving distress, ambiguity, or escalation. This role involves roleplaying and reviewing clinical scenarios with AI agents. As such, we are ideally seeking candidates who bring creative or performance-driven strengths , as these competencies enhance the realism, nuance, and emotional depth needed for AI safety testing. Examples of these can include, but are not limited to: Theatre degrees or studies Acting, theatre, improv, or voice-over experience Strong writing skills, especially dialogue or scenario writing Experience creating or inhabiting characters (e.g., performers, TTRPG roleplay, narrative designers) Conversational design, interaction writing, or scripted roleplay experience Participation in gaming, interactive storytelling, or digital communities where roleplay is common

Requirements

Professional experience in one or more of the following:
LLM red teaming or AI safety evaluation
Trust & safety, content moderation, or policy enforcement
AI/ML evaluation, annotation, or QA workflows
Conversational analysis or behavioral risk assessment
Work involving sensitive or high-stakes user interactions
Strong understanding of:
AI safety principles and common failure modes
Behavioral risk, escalation patterns, and edge-case handling
Mental health sensitivity, boundaries, and responsible AI behavior
How users express distress, confusion, or harmful intent in conversation
Ability to identify:
Safety violations and policy gaps
Missed or mishandled risk signals
Unsafe, misleading, or overconfident responses
Inappropriate tone or boundary-setting
Failures in escalation, de-escalation, or resolution
Inconsistencies across multi-turn interactions
Experience with or Interest in:
Red teaming methodologies and adversarial testing
Evaluating conversational AI systems or chatbots
Developing or applying evaluation frameworks and rubrics
Understanding how AI systems perform under real user behavior
Comfort with:
Tech tools and platforms (Slack, spreadsheets, dashboards)
Evaluating AI-generated responses (no coding required, but must be tech-comfortable)
Ambiguity, iteration, and feedback-driven workflows
Willingness to:
Sign NDAs and work with sensitive or high-impact content

Nice To Haves

Background in mental health, behavioral science, or psychology
Experience in QA, annotation, or qualitative analysis
Experience with AI systems in sensitive domains (e.g., healthcare, safety)
Familiarity with evaluation metrics or safety frameworks

Responsibilities

Designing and executing red-teaming scenarios across diverse user behaviors
Reviewing AI-generated responses for safety, accuracy, and policy compliance
Identifying failure modes, edge cases, and behavioral risks
Assessing whether AI appropriately recognizes and responds to distress or escalation
Evaluating tone, boundaries, and appropriateness in sensitive interactions
Detecting misleading, overconfident, or unsafe responses
Evaluating multi-turn conversations for consistency and risk handling
Identifying gaps in responses, including missed signals or incomplete handling
Conducting qualitative analysis to identify behavioral patterns and system weaknesses
Documenting edge cases, failure patterns, and safety risks
Applying or contributing to evaluation rubrics, taxonomies, and frameworks
Supporting quality assurance (QA) to ensure consistency across evaluations
Collaborating with internal teams on AI safety and evaluation improvements
Participating in red teaming exercises to surface system vulnerabilities
Maintaining strict confidentiality and quality standards