Conversational AI Agent Analyst

Roku•San Jose, CA

1d•$112,000 - $153,000•Hybrid

About The Position

Teamwork makes the stream work. Roku is changing how the world watches TV Roku is the #1 TV streaming platform in the U.S., Canada, and Mexico, and we've set our sights on powering every television in the world. Roku pioneered streaming to the TV. Our mission is to be the TV streaming platform that connects the entire TV ecosystem. We connect consumers to the content they love, enable content publishers to build and monetize large audiences, and provide advertisers unique capabilities to engage consumers. From your first day at Roku, you'll make a valuable - and valued - contribution. We're a fast-growing public company where no one is a bystander. We offer you the opportunity to delight millions of TV streamers around the world while gaining meaningful experience across a variety of disciplines. About the Team Our team drives the end-to-end tools, systems, and AI-powered experiences that enable world-class support for millions of Roku customers and the advisors who help them. We own the strategy, roadmap, and delivery of the platforms that power customer support at scale—including our CRM, contact center technologies, agent tools, knowledge systems, automation capabilities, and LLM-based customer service bots. About the Role Roku is seeking a Conversational AI Agent Analyst to join our Product Management team. In this role, you will focus on generating data-driven insights and guiding the roadmap for our customer-facing LLM-powered Bot products. You will analyze key performance metrics and customer feedback to identify opportunities for improving the bot experience. Your findings will directly influence product enhancements and overall customer satisfaction. If you have a background in business analysis/business automation/quality assurance using AI-driven tools, a keen analytical mindset, and excellent communication skills, we’d love to hear from you! For California Only - The estimated annual salary for this position is between $112,000 - $153,000 annually. Compensation packages are based on factors unique to each candidate, including but not limited to skill set, certifications, and specific geographical location. This role is eligible for health insurance, equity awards, life insurance, disability benefits, parental leave, wellness benefits, and paid time off.

Requirements

3+ years of experience in business analysis, product analytics, data analysis, or quality assurance, preferably working with customer-facing technology or AI-powered tools
Bachelor’s degree in a relevant field (e.g., Business, Data Science, Computer Science, or related discipline) or equivalent practical experience
Familiarity with conversational AI systems, LLMs, Bots, or virtual assistants, and an understanding of customer service metrics. Experience specifically with Bot performance analysis is a plus
Proven ability to analyze large datasets and derive meaningful insights. Comfortable working with metrics and KPIs to evaluate product performance. Not afraid to roll up your sleeves and manually analyze bot conversations.
Proficiency with Excel, BI tools, or data visualization software to analyze and report on performance trends. Experience writing SQL queries for data analysis is a plus (SQL is nice-to-have, but not a strict requirement)
Excellent written and verbal communication skills. Able to clearly present findings and recommendations to both technical and non-technical stakeholders. Strong storytelling skills with data – capable of crafting clear narratives around bot performance and customer experience for various audiences
Experience working in cross-functional teams and the ability to influence product decisions through data-driven storytelling
A proactive and curious mindset with strong problem-solving skills. Detail-oriented when diagnosing issues and evaluating the bot’s accuracy and effectiveness.
Enthusiasm for learning about the latest AI tools and techniques, and applying that knowledge to improve Roku’s conversational AI products continuously
Ability to thrive in a fast-paced environment and adapt as the product and business needs evolve
Excellent organization and time management to handle multiple analyses and reporting deadlines

Responsibilities

Analyze Bot Performance: Evaluate the performance of existing and new bot features by tracking and interpreting key bot KPIs, including Bot CSAT (Customer Satisfaction) scores, Bot Containment Rate, Bot Accuracy Rate, Bot Recontact Rate, and more. Identify trends, gaps, and areas for improvement in these metrics. Transform data and customer interaction logs into actionable insights. Provide a precise analysis of what’s driving bot success or failure
Drive Product Improvements: Work closely with product managers, customer service operations, and our 3rd party bot vendor to prioritize enhancements and fixes for the bot based on your analyses. Use your findings to shape the product roadmap and ensure development efforts align with user needs and business goals. Partner with engineering, design, customer support, and QA teams to communicate Bot performance results and customer pain points. Advocate for users by sharing data-backed stories that highlight opportunities to refine the conversational AI experience
Product Quality Ownership: Own creation of test cases and hands-on testing of any new bot releases. Work with cross-functional stakeholders to ensure test cases are executed, triaged, and driven to resolution. Work with internal stakeholders and our vendors to ensure bugs and enhancement requests are prioritized and completed on time
Reporting & Monitoring Improvements: Work with internal data science teams and our bot vendor to maintain and update dashboards in Looker as well as on our 3rd party bot system, to ensure they are providing necessary insights
Bot System Administration: Be the single point of contact for bot system administration tasks, ensuring system configurations are updated and optimized as needed