AI Quality Analyst
About The Position
We are seeking an AI Quality Analyst to own the full quality cycle for our AI-powered SMS bot and AI-graded call training system used by a private school network. Both systems utilize extensive test suites and a RAG-based knowledge base. This role is not a software engineering position, but rather focuses on operating and maintaining the existing infrastructure by editing JSON, writing Markdown, and running CLI scripts. The ideal candidate will have native-level English fluency for writing realistic SMS conversations and precise evaluation criteria, be comfortable with IDEs and terminal commands, possess extreme attention to detail, and be able to quickly learn a complex domain.
Requirements
- Native-level English fluency (non-negotiable) for writing realistic SMS conversations and precise pass/fail evaluation criteria.
- Comfortable editing JSON and Markdown in an IDE.
- Ability to run commands in a terminal without hand-holding.
- Extreme attention to detail.
- Ability to learn a complex domain quickly (e.g., education, state government programs).
Nice To Haves
- Experience writing test cases or QA documentation.
- Experience with chatbot QA, conversational AI testing, or LLM evaluation.
Responsibilities
- Review production conversations to identify and document bad bot responses.
- Create JSON test files to reproduce identified issues.
- Update and fix knowledge base content in Markdown files, including WRONG/RIGHT examples.
- Execute test suites to verify fixes and prevent regressions.
- Maintain and expand the knowledge base as the program evolves.
Benefits
- Ramping up hours as needed.
- Extensive internal documentation for learning.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Part-time
Career Level
Mid Level
Education Level
No Education Listed
