Evaluations Operations Analyst (Contract)

Cartesia • San Francisco, CA

5h • $25 - $35 • Remote

About The Position

Cartesia is a research lab looking for an Evaluations Operations Analyst to support their evaluation operations as they scale their audio model development. The analyst will work alongside the Evaluations Manager within the Product and Research Operations team, helping run the day-to-day operations for a globally distributed workforce. This role involves serving evaluation tasks to annotators, managing assignment distribution, tracking completion, and acting as a first line of support for evaluators. The position also requires maintaining project trackers, assignment records, and operational documentation, and supporting calibration workflows to ensure data quality.

Requirements

• Experience managing or coordinating annotator workforces, including task assignment, scheduling, and follow-up.
• Hands-on experience with data annotation platforms.
• Strong written communication skills, particularly in coordinating with globally distributed contractor workforces in different time zones and language groups.
• Ability to work independently and manage your own time effectively to meet tight project deadlines.
• Comfort working in spreadsheets, task management tools (e.g., Asana), and messaging platforms (e.g., Slack).

Nice To Haves

• Experience with linguistics or in language-related fields.
• Proficiency in languages beyond English to assist with multilingual evaluation workflows.
• Familiarity with human evaluation methodologies (e.g., scale-based ratings).
• Proficiency in using LLM assistants to move quickly without sacrificing quality.

Responsibilities

• Serve evaluation tasks to our annotator workforce and manage assignment distribution across projects and locales.
• Track and verify completion of assignments, flagging delays or issues before they become blockers.
• Act as a first line of support for evaluators, fielding operational questions about project setup, platform access, and task instructions.
• Help maintain project trackers, assignment records, and operational documentation.
• Support calibration workflows as needed to ensure evaluation data meets our quality bar.

Benefits

• Competitive base salary alongside attractive equity package.
• Fully covered medical insurance along with dental and vision for you and your family.
• 9 weeks paternity & 12 weeks maternity leave
• 401(k)
• A monthly stipend to help you get to and from the office.
• Take as much time as you need to recharge your batteries.
• Lunch, dinner and plenty of snacks, provided daily.
• Your own personal Yoshi

What This Job Offers

• Job Type: Part-time
• Career Level: Mid Level
• Education Level: No Education Listed