AI Voice Evaluation Specialist - 3 month contract

Innodata Inc.

2d•$20 - $25

About The Position

Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide data, evaluation frameworks, and human expertise for building trustworthy AI systems. This role is a 3-month, W2, full-time contract (40 hours per week) where specialists will evaluate AI model performance through voice-based conversations. The goal is to compare two AI models using the same scenario, identify which model provides a stronger conversational experience, and provide structured evaluations based on defined quality criteria.

Requirements

Bachelors Degree
Strong attention to detail and ability to notice subtle differences in conversational quality.
Excellent listening and comprehension skills.
Strong written communication skills, particularly the ability to explain observations clearly and objectively.
Ability to consistently follow detailed evaluation guidelines.
Comfort speaking naturally and role-playing different conversational scenarios.
Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
Strong critical-thinking and analytical skills.
Reliability and consistency when completing structured evaluation tasks.
Familiarity with AI assistants, voice-based AI, or conversational systems

Responsibilities

Review an assigned scenario and roleplay a natural conversation with two different AI models.
Conduct comparable conversations with Model A and Model B, keeping the scenario, conversational approach, and overall interaction as consistent as possible.
Maintain approximately the same number of conversational turns with each model to support a fair comparison.
Record your voice during each interaction and pay close attention to the quality and behavior of the model's responses.
Evaluate each model across five defined evaluation dimensions.
Identify and categorize relevant error clusters that may occur in the audio or conversation.
Compare the performance of Model A and Model B based on the evaluation criteria and your observations.
Select the model that demonstrated the stronger overall performance according to the defined evaluation framework.
Write a detailed rationale explaining your final preference, referencing specific examples and observations from both conversations.
Apply evaluation guidelines consistently across different scenarios and model interactions.