AI Quality Assurance Analyst - Remote
About The Position
OneDigital is seeking a detail-oriented and curious AI Quality Assurance Analyst to join our embedded product team. This role is responsible for ensuring that the AI-powered applications built by our product team are reliable, accurate, and ready for real-world use across OneDigital’s business lines. As AI becomes core to how we work and serve clients, quality assurance takes on a new dimension. This is not just about catching bugs. It is about evaluating how AI behaves: whether it responds accurately, handles edge cases gracefully, and delivers a consistent experience for users. You will partner closely with our Principal Engineer, AI Builders, and Product Managers to test both the AI output layer and the application functionality built around it. This is a hands-on individual contributor role for someone who is methodical, curious, and excited about the frontier of AI product development.
Requirements
- Experience in a QA, testing, or quality-focused role, with some exposure to AI or LLM based products preferred
- Strong ability to evaluate AI-generated outputs for quality, accuracy, and appropriateness
- Familiarity with prompt-driven systems and an intuition for how language models can fail or produce unexpected results
- Experience writing and executing test cases and documenting defects clearly
- Comfort working in no-code and low-code environments
- Strong written communication skills, with the ability to describe issues precisely and concisely
- Curiosity and adaptability in a fast-moving, emerging technology space
- Ability to work independently and manage multiple testing workstreams at once
- Bachelor’s degree or equivalent experience
- Prior experience in a QA role within a software, SaaS, or technology-adjacent environment
- Exposure to AI or LLM-based tools, either professionally or through independent exploration
- Familiarity with issue tracking tools (e.g., Jira, Linear, or similar)
- Understanding of AI governance, data privacy, and responsible AI considerations
Responsibilities
- Design and execute test plans for AI-powered applications across both output quality and functional behavior
- Evaluate AI responses for accuracy, consistency, hallucination, and edge case handling
- Test application workflows, UI behavior, and integrations across internal and client-facing tools
- Identify, document, and track defects using standard issue tracking tools (e.g., Jira or equivalent)
- Partner with AI Builders and Product Managers to define acceptance criteria and quality standards for AI features
- Develop and maintain repeatable test cases and QA documentation as AI applications evolve
- Conduct regression testing as applications are updated or new features are released
- Collaborate with the Principal Engineer and product team to surface quality risks early in the build process
- Contribute to defining what “good” looks like for AI output quality at OneDigital, helping shape internal QA standards for LLM-based applications
- Support testing across multiple business lines including Employee Benefits, Retirement, Wealth, and P&C
Benefits
- health, wellbeing, retirement, and other financial benefits
- paid time off
- overtime pay for non-exempt employees
- robust learning and development programs
- reimbursement of job-related expenses per the company policy
- employee perks and discounts
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
