Applied Scientist, Artificial General Intelligence, AGI Data Services

Amazon • Boston, MA

1d • $136,000 - $184,000 • Onsite

About The Position

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.

Requirements

• Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field
• Experience programming in Java, C++, Python or related language
• Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse

Nice To Haves

• Experience implementing algorithms using both toolkits and self-developed code
• Have publications at top-tier peer-reviewed conferences or journals

Responsibilities

• Collaborate closely with core scientist team developing Amazon Nova models.
• Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
• Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks.
• Perform expert-level manual audits.
• Conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities.
• Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
• Set up the configuration of data collection workflows.
• Communicate quality feedback to stakeholders.
• Enhance customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
• Support quality solution design.
• Conduct root cause analysis on data quality issues.
• Research new auditing methodologies.
• Find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards.
• Work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave
• sign-on payments
• restricted stock units (RSUs)

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level