GenAI Tuning Analyst

USAN•Norcross, GA

4d•Onsite

About The Position

As a GenAI Tuning Analyst, you will be responsible for the continuous improvement, accuracy, and "brand voice" of our Generative AI deployments within the Contact Center ecosystem, including agentic bots. You will work at the intersection of data science, linguistics, and customer operations to ensure our AI agents and agent-assist tools provide precise, empathetic, and compliant resolutions. This role requires a professional learner; a high-signal individual who pairs raw brainpower and adaptability with the hunger to master the evolving science of AI performance. Your goal is to transform generic LLM output into specialized, contact center domain-aware intelligence.

Requirements

1-2 years in AI/NLP, Data Analysis, high-level Contact Center Operations.
Analyzing Intent Accuracy to identify gaps in the AI’s natural language understanding (NLU/NLP).
Experience assisting in User Acceptance Testing for new AI features or tool rollouts.
Proficiency in prompt engineering techniques (Chain-of-Thought, Few-Shot, Zero Shot).
Basic understanding of LLM architectures (GPT-4, Claude, Llama).
Skilled at identifying trends in customer interaction data to suggest process improvements.
Exceptional written and verbal communication skills with an obsession over tone, grammar, and brand consistency.
Ability to act as a bridge between technical teams and frontline operations.
Moderate customer meetings to align with company direction on our AI tools.

Nice To Haves

Secondary experience in Quality Assurance is a plus.
Experience with RAG (Retrieval-Augmented Generation) frameworks.
Background in Linguistics or Cognitive Science.
Experience with data visualization tools (Tableau/ PowerBI) and basic SQL to pull interaction data.
Familiarity with Contact Center platforms (e.g., Amazon Connect Workspaces, Salesforce Service Cloud, Genesys, Nice, or HubSpot).

Responsibilities

Lead Reinforcement Learning from Human Feedback (RLHF) cycles. You will review AI-generated outputs and "grade" them based on accuracy, empathy, and adherence to compliance protocols.
Participate in Customer Tuning monthly meetings pertaining to a complete suite of USAN AI Product Specific deployments.
Define and track AI-specific KPIs, such as Hallucination Rate, Refusal Rate, and Intent Accuracy, alongside traditional metrics like First Contact Resolution (FCR).
Implement and monitor safety filters to ensure the AI remains neutral, avoids prohibited topics, and protects PII (Personally Identifiable Information).
Partner with Subject Matter Experts (SMEs) in Legal, Compliance, and Operations to translate business policy into technical model constraints.
Iterate on complex system prompts to guide LLM behavior for specific contact center use cases (e.g., automated summaries, live chat responses, and knowledge base retrieval).
Identify and curate high-quality "golden datasets" from historical call transcripts and chat logs to be used for fine-tuning and few-shot prompting.
Translate customer feedback into actionable product improvements by gathering input, researching root causes, writing clear technical requirements for developers, and analyzing post-update results (e.g., for agent scorecards/GenAI tools) before returning changes to the customer.