Annotation Project Specialist

American IT Systems•Redmond, WA

5d•Hybrid

About The Position

This role involves overseeing annotation projects, ensuring high-quality data delivery, and managing vendor performance. The specialist will be responsible for guideline implementation, quality assurance, training, and communication with various stakeholders.

Requirements

3+ years experience in annotation project management roles.
Proven ability to write clear annotation guidelines and define KPIs for quality measurement.
Experience training raters or annotators on guidelines and quality standards.
Strong technical aptitude to navigate annotation UIs and communicate effectively with software engineers about pipeline issues.
Excellent written and verbal communication skills, including the ability to document processes and report metrics clearly.
Experience coordinating with vendors and cross-functional teams to ensure project success.
Ability to work independently in a fast-paced, ambiguous environment.

Nice To Haves

Background or familiarity with software engineering concepts and troubleshooting.

Responsibilities

Oversee annotation vendor performance, ensuring timely and high-quality data deliveries.
Track and report on vendor success metrics.
Coordinate onboarding & certification of new raters.
Ensure correct implementation of annotation guidelines.
Partner with researchers to develop, refine, and update annotation guidelines.
Uphold the researcher quality bar and high data quality standards.
Oversee DQA (Double Quality Assurance) processes and success.
Develop, review, and improve quality assurance processes.
Manage DQA resources and reporting.
Organize and deliver annotator training sessions.
Coordinate with annotation vendors for ongoing training and calibration.
Report on annotation metrics and quality trends.
Provide monthly written updates.
Communicate expectations, progress, and changes with research and vendor stakeholders.