Conversational AI Bot Analyst

Roku•San Jose, CA

21d•$112,000 - $153,000•Hybrid

About The Position

Roku is seeking a Conversational AI Bot Analyst to join our Product Management team. In this role, you will focus on generating data-driven insights and guiding the roadmap for our customer-facing LLM-powered Bot products. You will analyze key performance metrics and customer feedback to identify opportunities for improving the bot experience. Your findings will directly influence product enhancements and overall customer satisfaction. If you have a background in business analysis/business automation/quality assurance using AI-driven tools, a keen analytical mindset, and excellent communication skills, we’d love to hear from you!

Requirements

3+ years of experience in business analysis, product analytics, data analysis, or quality assurance, preferably working with customer-facing technology or AI-powered tools
Bachelor’s degree in a relevant field (e.g., Business, Data Science, Computer Science, or related discipline) or equivalent practical experience
Familiarity with conversational AI systems, LLMs, Bots, or virtual assistants, and an understanding of customer service metrics.
Proven ability to analyze large datasets and derive meaningful insights. Comfortable working with metrics and KPIs to evaluate product performance. Not afraid to roll up your sleeves and manually analyze bot conversations. Proficiency with Excel, BI tools, or data visualization software to analyze and report on performance trends. Experience writing SQL queries for data analysis is a plus (SQL is nice-to-have, but not a strict requirement)
Excellent written and verbal communication skills. Able to clearly present findings and recommendations to both technical and non-technical stakeholders. Strong storytelling skills with data – capable of crafting clear narratives around bot performance and customer experience for various audiences
Experience working in cross-functional teams and the ability to influence product decisions through data-driven storytelling
A proactive and curious mindset with strong problem-solving skills. Detail-oriented when diagnosing issues and evaluating the bot’s accuracy and effectiveness. Enthusiasm for learning about the latest AI tools and techniques, and applying that knowledge to improve Roku’s conversational AI products continuously
Ability to thrive in a fast-paced environment and adapt as the product and business needs evolve
Excellent organization and time management to handle multiple analyses and reporting deadlines

Nice To Haves

Experience specifically with Bot performance analysis is a plus
Experience writing SQL queries for data analysis is a plus (SQL is nice-to-have, but not a strict requirement)

Responsibilities

Analyze Bot Performance: Evaluate the performance of existing and new bot features by tracking and interpreting key bot KPIs, including Bot CSAT (Customer Satisfaction) scores, Bot Containment Rate, Bot Accuracy Rate, Bot Recontact Rate, and more. Identify trends, gaps, and areas for improvement in these metrics. Transform data and customer interaction logs into actionable insights. Provide a precise analysis of what’s driving bot success or failure
Drive Product Improvements: Work closely with product managers, customer service operations, and our 3rd party bot vendor to prioritize enhancements and fixes for the bot based on your analyses. Use your findings to shape the product roadmap and ensure development efforts align with user needs and business goals. Partner with engineering, design, customer support, and QA teams to communicate Bot performance results and customer pain points. Advocate for users by sharing data-backed stories that highlight opportunities to refine the conversational AI experience
Product Quality Ownership: Own creation of test cases and hands-on testing of any new bot releases. Work with cross-functional stakeholders to ensure test cases are executed, triaged, and driven to resolution. Work with internal stakeholders and our vendors to ensure bugs and enhancement requests are prioritized and completed on time
Reporting & Monitoring Improvements: Work with internal data science teams and our bot vendor to maintain and update dashboards in Looker as well as on our 3rd party bot system, to ensure they are providing necessary insights
Bot System Administration: Be the single point of contact for bot system administration tasks, ensuring system configurations are updated and optimized as needed