Senior Analyst Conversational AI Analytics

National Debt Relief, LLC.

2d•$114,500 - $131,500•Remote

About The Position

National Debt Relief is looking for a Senior Analyst, Conversational AI Analytics to join the Digital & CX analytics team. This role sits at the intersection of data, user experience (UX), and Agentic AI - you will be the analytical engine behind our conversational AI experiences, translating how real users interact with our voice and chat AI into insights that shape better conversations, smarter flows, and stronger outcomes. You are not the engineer who builds AI Agents, but you are essential to making them work well. You will own the measurement framework for our conversational AI products - defining what good looks like, tracking where experiences break down, and partnering with designers, product managers, and our Applied AI team to close the gap between current performance and the ideal user journey. This is a high-impact, cross-functional role with direct exposure to senior leadership. The right person brings both analytical rigor and a genuine curiosity for how people communicate with AI.

Requirements

4+ years of experience in UX analytics, product analytics, or a related data-focused role, with at least 1–2 years working on conversational AI, chatbot, or voice AI products.
Demonstrated ability to analyze conversation-level data - transcript review, intent classification accuracy, fallback rates, CSAT/sentiment scoring - and turning findings into design or product recommendations.
Proficiency with analytics and BI tools (e.g., Tableau, Looker, Power BI, or similar) for building dashboards and self-serve reporting.
Comfortable working with SQL to query conversation logs and event data; Python or R a plus but not required.
Strong understanding of UX research methods - usability testing, user journey mapping, qualitative coding of transcripts, and mixed-methods synthesis.
Familiarity with core conversational AI concepts - intents, entities, dialogue management, NLU/NLP, fallback handling, escalation logic - sufficient to contribute meaningfully to cross-functional discussions without owning the engineering.
Excellent communication skills; able to distill complex data into clear narratives for both technical teammates and non-technical stakeholders.
Comfortably operating with ambiguity and managing multiple workstreams simultaneously in a fast-moving environment.
Computer competency and ability to work with a computer.
Prioritize multiple tasks and projects simultaneously.
Exceptional written and verbal communication skills.
Punctuality expected, ready to report to work on a consistent basis.
Attain and maintain high performance expectations on a monthly basis.
Work in a fast-paced, high-volume setting.
Use and navigate multiple computer systems with exceptional multi-tasking skills.
Remain calm and professional during difficult discussions.
Take constructive feedback.

Nice To Haves

Experience with conversational analytics or NLP platforms (e.g., Qualtrics Conversational Analytics, Cognigy.AI, Google CCAI Insights, Gong, or similar).
Familiarity with voice AI or IVR platforms (LiveKit, Sierra, Parloa, Replicant, or similar) from a measurement and QA perspective.
Direct experience with LLM-based products, prompt evaluation, or retrieval-augmented generation (RAG) quality assessment.
Background in fintech, financial services, or debt relief / credit counseling.
Experience with A/B testing and experimentation frameworks applied to conversational flows.
Formal UX research certification or training (Nielsen Norman Group, UXQB, or similar).

Responsibilities

Own the measurement framework for conversational AI products - defining KPIs across containment, resolution, deflection, intent recognition accuracy, conversation completion rates, drop-off, and sentiment.
Build and maintain dashboards that surface conversation health metrics, user behavioral patterns, and performance trends across voice and chat channels.
Provide UX insights via analysis of conversation transcripts, dialogue flows, and session data to identify friction points, misunderstood intents, fallback loops, and abandonment patterns.
Partner with the Applied AI team to establish baseline performance benchmarks and define what success looks like for each conversational initiative.
Translate raw interaction data into clear, actionable recommendations for conversation designers, product managers, and ML teams.
Conduct mixed-methods research - combining quantitative conversation analytics with qualitative usability testing, session replays, and user feedback - to build a complete picture of user experience quality.
Map conversation flows against actual user journeys to identify where AI-guided paths diverge from user expectations or intent.
Support the design and analysis of A/B tests on conversational flows, dialogue variations, and escalation triggers.
Synthesize research findings into user journey documentation, annotated flow diagrams, and prioritized insight reports for product and design stakeholders.
Serve as the data-informed voice in conversation design reviews - bringing evidence from real user interactions to design critiques, sprint reviews, and roadmap discussions.
Present insights and performance reports to senior stakeholders in clear, non-technical language that connects conversational metrics to business outcomes (conversion rates, customer satisfaction, operational cost savings).
Partner with engineering to ensure proper logging, tagging, and event tracking are in place to support robust conversation analytics.
Collaborate with the broader product organization to align conversational AI metrics with company-wide data governance and reporting standards.
Establish a regular cadence of conversation audits - systematically reviewing transcript samples to surface emergent issues before they appear in aggregate metrics.
Monitor model and flow performance post-deployment, flagging regression or drift and coordinating response with the AI and engineering teams.
Stay current with trends in conversational AI, NLP evaluation methods, and UX analytics tooling - and identify where new capabilities can strengthen the team’s analytical practice.