LLM - AI Quality Analyst (Personalization) - Portuguese

Careerflow.ai • ,

2d • Remote

About The Position

We're looking for sharp and detail-oriented analysts to join a global team evaluating a personalization feature for a leading AI assistant. In this role, you'll put the AI through its paces. Testing how well it uses real personal context from past conversations, Gmail, Google Search, and YouTube activity to give genuinely helpful, tailored responses. This isn't a passive review job. You'll be designing your own prompts drawn from your actual experiences, then rigorously evaluating the AI's responses across multiple quality dimensions. Think of yourself as part product tester, part quality critic.

Requirements

• Strong written and reading proficiency in Portuguese, this is the primary language for the project.
• Exceptional analytical thinking, particularly when it comes to evaluating nuanced or ambiguous AI-generated content.
• Willing to use your primary personal Google account (not a test account) and enable personal data sources for evaluation
• Ability to design creative, context-rich prompts based on your own personal experiences.
• A keen eye for spotting subtle differences between model responses, things like over-narrating, forced connections, or weak personalization.
• Strong written communication skills. You'll need to write clear, defensible evaluation notes regularly.
• Comfortable working independently and staying self-directed in a fully remote setup.
• Reliable desktop or laptop with a solid internet connection.
• Full-time availability in your local time zone, with 4 hours of daily overlap with PST.

Nice To Haves

• Prior experience in data annotation, AI quality evaluation, content moderation, or a related field.
• A BS/BA degree (or equivalent experience) in a field like Linguistics, Journalism, Computer Science, Policy, Law, Ethics, or another analytically rigorous discipline.
• Familiarity with personalization concepts and an intuition for when AI responses feel genuine versus when they seem off.

Responsibilities

• Design and run multi-turn conversational prompts (typically 1-5 turns) that challenge the AI to draw on your personal data and experiences.
• Assess whether the model understood your intent and applied personalization appropriately, or where it missed the mark.
• Evaluate Grounding: flag any claims the model makes about you that aren't supported by evidence, including hallucinations or flawed inferences.
• Evaluate Integration: check whether personal data is woven into responses naturally, or whether the AI comes across as robotic or over-explanatory.
• Conduct side-by-side (SxS) comparisons of two model responses, ranking them on helpfulness, usability, and overall quality.
• Write clear, structured rationales for your rankings citing specific turns in the conversation to back up your assessments.
• Verify "Debug Info" to confirm that chat summaries and external data sources were properly referenced.
• Keep your evaluation data clean by deleting test conversations after each session so they don't influence future results.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level
• Education Level: Associate degree