Agentic Evaluation Scientist, Amazon Customer Service

Amazon • Seattle, WA

1d • $142,800 - $193,200 • Onsite

About The Position

Amazon's Customer Service (CS) is undergoing a major transformation leveraging agentic AI techniques to investigate why customers experience defects and how to prevent them in the future. We're looking for scientists to help drive the quality and capabilities of these agentic systems. Amazon's Customer Experience Improvement (CXI Tech) team is looking for scientists to help drive rigorous development in the agentic era. We seek to identify and eliminate unsatisfactory purchasing experiences that may cause customers to return items or not return to Amazon. This role requires you to re-imagine the way we leverage customer interactions/feedback for defect identification and the actions we take for reliable downstream defect resolution for an improved customer experience throughout the consumer business. Driven by the vision that no defect need ever impact customers twice, we work to develop rigorously tested AI systems to investigate what caused individual defects experienced by customers, and identify the needed changes to ensure they do not occur again. In particular, we are seeking scientists with a proven track record of making efficient use of human annotations to evaluate AI systems. Come help Amazon ensure the best possible experience for our customers!

Requirements

• 3+ years of building models for business application experience
• PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
• Experience in patents or publications at top-tier peer-reviewed conferences or journals
• Experience programming in Java, C++, Python or related language
• Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing

Nice To Haves

• Experience using Unix/Linux
• Experience in professional software development
• Experience building machine learning models or developing algorithms for business application

Responsibilities

• Develop evaluation techniques for agentic solutions that operate at Amazon scale developing techniques to make efficient use of human annotations to establish rigorous performance of agentic systems.
• Develop tools and patterns that improve the performance of agentic investigation systems.
• Create standalone ML powered models when needed to support automated investigation of customer defects.
• Develop techniques to measure and improve the performance of agentic systems, even when the performance of those systems exceed the existing human benchmark
• Interface closely with Engineering and Product partners to align the direction of scientific development with needs of the greater business
• Publish techniques internally and externally (when possible) to promote greater rigor in agentic development

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave
• sign-on payments
• restricted stock units (RSUs)

What This Job Offers

• Job Type: Full-time
• Career Level: Senior