Applied Science Manager, Artificial General Intelligence

Amazon • Boston, MA

2d

About The Position

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Science Manager with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to lead a team ensuring the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.

As part of the AGI team, an Applied Science Manager will lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. The manager will guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models. The Applied Science Manager will oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team. A critical aspect of this role involves managing the development and maintenance of LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Science Manager will also oversee the configuration of data collection workflows and ensure effective communication of quality feedback to stakeholders. The manager will have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.

The Applied Science Manager will be responsible for recruiting, hiring, and developing team members, conducting performance reviews, setting clear expectations and growth plans, and fostering a culture of scientific excellence and innovation. The manager will communicate with senior leadership, cross-functional technical teams, and customers to collect requirements, describe product features and technical designs, and articulate product strategy.

An Applied Science Manager with the AGI team will lead quality solution design, guide root cause analysis on data quality issues, drive research into new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. The manager will work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice. The manager will also conduct regular 1:1s with team members, provide mentorship and coaching, and ensure the team delivers high-impact results aligned with organizational goals.

Requirements

• 2+ years of team management experience
• Master's degree in engineering, technology, computer science, machine learning, robotics, operations research, statistics, mathematics or equivalent quantitative field
• 2+ years of machine learning, statistical modeling, data mining, and analytics techniques experience
• 2+ years of building machine learning models or developing algorithms for business application experience
• 3+ years of programming in Java, C++, Python or related language experience

Nice To Haves

• Experience communicating with users, other technical teams, and management to collect requirements, describe software product features, and technical designs
• 3+ years of scientists or machine learning engineers management experience
• Ph.D. in engineering, technology, computer science, machine learning, robotics, operations research, statistics, mathematics or equivalent quantitative field
• Have publications at top-tier peer-reviewed conferences or journals
• 4+ years of solving business problems through machine learning, data mining and statistical algorithms experience
• Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution, or experience in developing and deploying LLMs in production on GPUs, Neuron, TPU or other AI acceleration hardware

Responsibilities

• Lead and mentor a team of Applied Scientists
• Develop comprehensive quality strategies and auditing frameworks
• Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies
• Oversee expert-level manual audits and meta-audits
• Manage the development and maintenance of LLM-as-a-Judge systems
• Oversee the configuration of data collection workflows
• Ensure effective communication of quality feedback to stakeholders
• Recruit, hire, and develop team members
• Conduct performance reviews
• Set clear expectations and growth plans
• Foster a culture of scientific excellence and innovation
• Communicate with senior leadership, cross-functional technical teams, and customers
• Lead quality solution design
• Guide root cause analysis on data quality issues
• Drive research into new auditing methodologies
• Optimize data quality
• Provide mentorship and coaching

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Manager