Lead QA Engineer

Alivo•Portland, ME

About The Position

Alivo is seeking a highly detail-oriented QA Engineer to take ownership of testing, release validation, and AI conversation quality across the Alivo platform. This role involves developing and running automated test suites, manually validating new features, identifying edge cases, evaluating user experience, and ensuring releases meet high standards. A key aspect of this role is evaluating the quality of AI-driven phone and chat conversations to ensure natural, accurate, and effective communication, while identifying opportunities for improvement. The QA Engineer will serve as the final quality gate before releases and help define standards for AI-powered customer communication. This position is ideal for individuals who thrive in fast-paced environments, enjoy proactive problem-solving, and possess strong quality instincts.

Requirements

2–5+ years of QA, software testing, product operations, or similar experience
Strong attention to detail and a methodical approach to testing
Excellent written and verbal communication skills
Ability to identify edge cases, failure modes, and customer experience issues
Strong problem-solving and analytical thinking skills
Experience creating clear bug reports and testing documentation
Comfortable working in a fast-paced, evolving startup environment
Highly detail-oriented and naturally curious
A strong critical thinker who enjoys finding problems before customers do
Comfortable making judgment calls about quality and user experience
Organized, reliable, and highly accountable
Able to communicate findings clearly and objectively
Passionate about creating exceptional customer experiences

Nice To Haves

Experience evaluating AI, LLM, or conversational AI systems
Experience with test management or bug tracking tools such as Linear, Jira, or TestRail
Experience with API testing tools such as Postman
Basic SQL, scripting, or technical troubleshooting skills
Experience in customer-facing, sales, or support roles
Exposure to AI-powered products or machine learning systems
Background in SaaS products or high-growth startup environments

Responsibilities

Own Product Testing
Develop, run, and maintain automated test suites
Manually test new features, workflows, and user experiences before release
Perform exploratory testing to identify bugs, usability issues, and edge cases
Validate functionality across staging and production-like environments
Execute structured test plans and release checklists to ensure consistent quality
Verify bug fixes and regression testing across product releases
Drive Product Quality & Release Readiness
Review feature implementations and release candidates before deployment
Partner with Engineering to identify risks and improve release confidence
File clear, reproducible bug reports with detailed documentation
Help establish and improve manual testing processes as the company grows
Serve as a trusted quality gate for production releases
Own AI Conversation Quality
Evaluate AI-powered phone and chat conversations for quality and effectiveness
Assess whether conversations are natural, accurate, helpful, and aligned with customer goals
Review objection handling, conversational flow, and customer experience outcomes
Identify failure modes, edge cases, hallucinations, and off-script behavior
Help define quality standards and acceptance criteria for AI-powered interactions
Build AI Evaluation Processes
Develop and maintain conversation evaluation datasets and testing scenarios
Create scoring rubrics and quality frameworks for measuring AI performance
Track AI quality over time and identify regressions or performance changes
Provide actionable recommendations that improve prompts, workflows, and AI behavior
Collaborate Cross-Functionally
Work closely with Engineering, Product, Customer Success, and Leadership
Provide feedback that improves both product quality and customer outcomes
Help prioritize quality improvements based on customer impact and business goals
Support a culture of quality throughout the organization