Personal Account Generalist Raters (Canada)
About The Position
Turing is seeking Personal Account Generalist Raters in Canada to participate in Gemini Personalization Evals. Raters perform generalist evaluation work using personalized prompts designed to retrieve context and data from connected Google apps. The role evaluates the quality of personalized AI interactions, including model performance, context retrieval, and response quality across connected apps. The work involves reviewing AI-generated responses based on professional email history and business account activity, assessing relevance, accuracy, helpfulness, and personalization, and providing detailed feedback to support model improvement. Raters must follow project guidelines and maintain data privacy and confidentiality. The engagement is contractor work for up to 16 weeks, requiring at least 4 hours per day and up to 40 hours per week, with 4 hours of overlap with PST. The role is remote and requires a desktop or laptop and reliable internet access.
Requirements
- Must currently be based in Canada.
- Must be willing and able to connect Google apps, such as Gmail, Google Photos, Google Calendar, and Google Drive, to Gemini for data retrieval and personalized prompt evaluations.
- Active use of standard Google workspace or consumer apps, with sufficient personal data and history to generate and evaluate retrieval prompts.
- Exceptional analytical thinking and ability to evaluate nuanced AI responses and identify strengths and weaknesses in personalization quality.
- Strong attention to detail when reviewing AI-generated responses and identifying subtle issues or inconsistencies.
- Excellent written communication skills for clear, structured evaluation feedback.
- Self-motivated and able to work independently in a remote environment.
- Desktop or laptop with a reliable internet connection.
- Bachelor's degree or equivalent practical experience in any field.
- Willingness to grant necessary app integration permissions and complete standard consent forms and non-disclosure agreements.
- Availability for at least 4 hours per day and up to 40 hours per week, including 4 hours of overlap with PST.
Nice To Haves
- Experience in AI evaluation, data annotation, content review, quality assurance, or a related analytical role is preferred but not required.
Responsibilities
- Evaluate AI-generated responses based on professional email history and business account activity.
- Assess whether responses are relevant, accurate, helpful, and appropriately personalized.
- Identify incorrect personalization, unsupported assumptions, and irrelevant recommendations.
- Compare multiple AI responses and determine which offers the better overall user experience.
- Provide clear, detailed, and well-reasoned evaluation feedback to support model improvements.
- Follow project guidelines to ensure consistent, accurate, and high-quality evaluations.
- Maintain data privacy and comply with project confidentiality requirements.
