AI Conversation Research & Evaluation Specialist

Gramian Consulting Group

About The Position

Gramian Consultancy is seeking experienced subject matter experts in psychology, social work, counseling, qualitative research, human-computer interaction, communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. The role involves reviewing conversations between people and AI models, assessing whether sufficient context was gathered before recommendations were made, identifying missed opportunities for clarification, and providing structured feedback to improve conversational quality. The ideal candidate has at least 3–5 years of relevant professional experience, strong analytical skills, and the ability to distinguish information that could materially change a recommendation from details that are less relevant. Candidates must have at least 3–5 years of relevant direct professional experience in one or more of the listed domains. The advisory-fields pathway requires at least five years of direct advisory or client-facing experience.

Requirements

At least 3–5 years of relevant direct professional experience in one or more of the specified domains.
Strong analytical skills and the ability to distinguish information that could materially change a recommendation from less relevant details.
For the licensed social work pathway: an MSW or equivalent graduate-level qualification; an LCSW, LICSW, LMSW, or equivalent applicable license or credential; and experience in client assessment, case review, interviewing, counseling, or related social-work practice.
For the counseling, therapy, or psychology pathway: a relevant master's degree with an LMFT, LPC, LCPC, or equivalent counseling or therapy license or credential, or a PhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology; plus relevant counseling, therapy, psychological assessment, interviewing, case review, or other client-facing professional experience.
For the qualitative research, HCI, or communication pathway: an MA, PhD, or equivalent advanced expertise in Human-Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline; experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, or qualitative coding; and ability to analyze conversations systematically and identify missing context, communication patterns, and opportunities for clarification.
For the mediation, coaching, student services, organizational consulting, or related advisory pathway: a relevant professional qualification, credential, or demonstrated subject-matter expertise; at least five years of direct advisory or client-facing experience; experience in coaching, mediation, student advising, organizational consulting, or comparable professional work; and demonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations.

Responsibilities

Review AI conversations involving real-world interpersonal situations and everyday decision-making.
Identify missing context and recommend relevant follow-up questions that could improve the quality of AI responses.
Evaluate whether gathering additional information would materially change an AI-generated recommendation.
Distinguish between load-bearing information that affects a decision and non-essential or merely descriptive details.
Conduct qualitative reviews and document findings using structured evaluation guidelines.
Assess whether conversations adequately consider relevant personal, psychological, social, financial, practical, and organizational factors.
Identify missed opportunities for clarification, deeper understanding, and information gathering.
Review conversations involving relationships, work, education, family, wellbeing, planning, and resource constraints.
Collaborate with research teams to ensure consistent, evidence-based assessments.
Maintain high standards of accuracy, clarity, and consistency while meeting project deadlines.