AI Quality Engineer

Phoenix Oversight Group LLC

3d • Remote

About The Position

PhoenixTeam is seeking an AI Quality Engineer to support a federal customer. The AI Quality Engineer will help design and implement AI-enabled solutions, develop and integrate AI capabilities, and ensure that skills, agents, and AI-generated prototypes meet quality, security, and compliance standards as they move toward production. This work supports broader modernization goals: automating and streamlining servicing operations, reducing manual processing, improving data quality, aligning with the customer's modernization standards, and establishing a roadmap toward the eventual retirement of legacy platforms.

Requirements

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent professional experience.
• Demonstrated experience in software quality assurance, test engineering, or a related discipline, including test plan design and defect management.
• Hands-on experience testing or validating AI-generated code, prototypes, or “vibe-coded” solutions for accuracy and reliability.
• Experience with automated testing tools and frameworks (e.g., Selenium) across web and enterprise applications.
• Experience with one or more of the following: Salesforce, AWS (including Lambda and Redshift), Java, or PostgreSQL, or a comparable enterprise technology stack.
• Familiarity with CI/CD pipelines and quality gates within modern software delivery practices.
• Strong analytical skills, with the ability to identify security, data-sensitivity, and compliance risks in AI-assisted solutions.
• Excellent written and verbal communication skills, with the ability to work across product, engineering, data, and business SME teams.
• Must be a U.S. citizen and eligible to obtain and maintain a Public Trust clearance / favorable suitability determination.

Nice To Haves

• Direct experience supporting Federal clients
• Experience with FedRAMP authorization processes
• Mortgage, fintech, or other regulated industry experience (preferred, not required — willingness to learn quickly is essential).
• Familiarity with AI safety, governance, and compliance.
• Ability to use or teach tools such as ChatGPT, Claude, Gemini, Notion, Airtable, Zapier, LangChain, or similar.
• Experience in startups, consulting, internal strategy, learning & development (L&D), or innovation teams.

Responsibilities

• Design and implement AI-enabled solutions for customer modernization use cases, developing and integrating AI capabilities into existing systems and workflows.
• Establish and enforce a quality assurance framework — test plans, acceptance criteria, and regression suites — for validating AI-generated code, skills, and agents before they are promoted through the innovation pipeline.
• Guide the development of reusable AI skills and agents for customer modernization tasks, with particular emphasis on automated testing, security scanning, data sensitivity, and scalability.
• Collaborate with DevOps engineers to create and orchestrate long-running, multi-agent workflows, ensuring reliability, observability, and recoverability in production environments.
• Support creation of the innovation pipeline that moves AI-generated (“vibe-coded”) prototypes from sandbox concepts into secure, enterprise-grade applications.
• Support Value Engineers' efforts by reviewing and quality-checking prototype outputs for accuracy, reliability, and adherence to standards prior to pipeline promotion.
• Continuously monitor AI-assisted development outputs for accuracy, reliability, and compliance drift, feeding findings back to Value Engineers and AI Engineers to improve prompts, skills, and agents over time.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level