Generalist - English & Hindi

Weekday AI

8d•$12•Remote

About The Position

This role supports the development of high-quality, reliable conversational AI systems by improving how large language models (LLMs) respond to real-world user queries. The focus is on evaluating and enhancing general chat behavior to ensure responses are accurate, clear, and aligned with human expectations across a wide range of topics and use cases.

Requirements

Bachelor’s degree in any discipline.
Native-level fluency in Hindi (ILR 5 / CEFR C2) and strong proficiency in English.
Hands-on experience using large language models (LLMs) and understanding their real-world applications.
Excellent writing skills with the ability to provide clear, structured, and nuanced feedback.
Strong attention to detail with the ability to identify subtle issues in content.
Comfortable working across diverse topics, domains, and requirements.
Background in fields requiring structured analytical thinking such as research, analytics, policy, linguistics, or engineering.
Strong college-level mathematics and reasoning skills.

Nice To Haves

Experience with RLHF (Reinforcement Learning from Human Feedback), model evaluation, or data annotation.
Background in content writing, editing, or quality review.
Experience comparing multiple outputs and making detailed qualitative judgments.
Familiarity with evaluation rubrics, benchmarks, or quality scoring frameworks.

Responsibilities

Evaluate AI-generated responses for accuracy, relevance, and effectiveness in answering user queries.
Perform fact-checking using trusted public sources and external tools.
Create high-quality evaluation data by identifying strengths, weaknesses, and factual inaccuracies in responses.
Assess reasoning quality, clarity, tone, and completeness of outputs.
Ensure responses align with expected conversational standards and system guidelines.
Apply consistent annotations using defined taxonomies, benchmarks, and evaluation frameworks.