Quality Analyst II

HighLevel

19h•Remote

About The Position

HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. We are looking for a Quality Analyst II to join Customer Success — Programs & AI Innovation as an evaluator-first specialist focused on the quality of AI-powered workflows that support our advisors before, during and after every customer call. You will spend most of your time evaluating AI output across ORA, our internal AI platform — reviewing pre-call briefs, in-call guidance, post-call grading and transcription-driven customer follow-ups. You will classify failure modes, document patterns and feed structured findings into the team that builds inside ORA. You will also do light hands-on work inside ORA — refining prompts, updating knowledge base entries and tweaking existing skills — closing the loop from “found an issue” to “shipped a fix.” This is a role with a deliberate growth path. You start in an evaluator-first lane with light creation work. As you build judgment, exceed expectations and demonstrate strong ownership, your scope expands toward deeper creation work alongside the Lead Analyst on the team.

Requirements

Insatiable curiosity and a self-learning mindset — this is non-negotiable. The AI space changes weekly and we hire people who level up on their own without waiting for permission or perfect instructions.
2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role where you reviewed and improved customer-facing or advisor-facing work
Working understanding of how LLMs behave — prompt design, common failure modes, conversational AI patterns
General understanding of databases and how data flows between systems
Comfort reading code and making minor edits — enough to follow integrations, debug prompt behavior, iterate inside ORA and hold informed technical conversations with engineering. You do not need to be a full-stack engineer.
Strong written communication and attention to detail — you can document a failure mode so clearly that an engineer can act on it without follow-up
Ability to follow structured guidelines and scoring rubrics consistently while suggesting improvements to them
Comfort working with evolving tools and AI-assisted workflows
Self-starter mentality — you can manage review volume, prioritize what matters and move work forward with minimal direction
Bachelor’s degree in a related field, or equivalent practical experience

Nice To Haves

Hands-on experience with AI development platforms such as Claude, Cursor or similar tools
Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture
Familiarity with SQL or similar database querying
Experience with APIs, webhooks or data integration between systems
Background in customer success, support, sales enablement or call center environments
Experience with the HighLevel platform or comparable SaaS products
Prior experience evaluating AI-generated conversations, chatbots or automated support output

Responsibilities

Evaluate AI Quality Across the Advisor Stack: You are the quality owner for advisor-facing AI inside ORA. You review AI output across the full call lifecycle and surface what is working, what is broken and where the bar needs to be raised.
Review pre-call briefs for accuracy, completeness and signal quality — customer context, adoption signals, retention flags
Evaluate in-call guidance — next-best-action prompts, suggestions and recommendation flows — for relevance and accuracy in live conversation
Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals
Audit transcription-driven follow-ups — customer updates, recap notes, follow-up tasks — for accuracy, tone and customer impact
Classify issues by type and severity — hallucination, missing context, wrong action, tone mismatch, KB gap — using standardized rubrics
Participate in calibration sessions to keep scoring consistent across reviewers
Iterate Inside ORA — Light Creation Work: You close the loop. When you spot an issue you can fix without engineering involvement, you fix it inside ORA.
Refine and iterate on prompts within existing skills and sub-agents
Update knowledge base entries to close information gaps surfaced in reviews
Tweak existing skills and orchestrations to improve output quality
Test changes, validate improvements and roll out updates inside ORA
Document what you changed and why — every iteration is a data point
Surface Trends & Drive Continuous Improvement: You connect individual evaluations to systemic insights. Leadership and the Lead Analyst rely on you to know where the next investment should land.
Tag conversations by product area, severity and issue type with consistent classification
Maintain a clear, trusted log of issue types, trends and conversation behaviors
Surface recurring patterns and systemic gaps to leadership and the Lead Analyst
Recommend prioritized improvement opportunities — what to fix, what to build, what to redesign
Build reporting that gives Success leadership visibility into AI quality and the improvement backlog
Partner with RevOps Engineering & the Lead Analyst: When a fix requires ORA platform changes or end-to-end design, you stay close.
Bring structured findings, repro examples and quality criteria when a platform change is needed
Partner with the Lead Analyst on end-to-end builds — bring evaluation signal, test the output, validate the bar
Co-define quality criteria for new sub-agents, skills and orchestrations before they ship
Stay close through the build to ensure improvements land as intended
Stay Sharp: Learn, Experiment, Share: The AI space moves weekly. We hire people who keep up on their own and bring what they learn back.
Stay current on LLM behavior, prompt patterns and AI evaluation techniques
Run small experiments with new prompts, tools or evaluation approaches inside ORA
Share what you learn with the broader Success team and the Lead Analyst