QA Analyst

Inizio Evoke • Tempe, AZ

9h

About The Position

The AI QA Analyst is responsible for validating the quality, reliability, and trustworthiness of outcomes produced by an agentic AI platform operating across both ad-hoc and structured work modes. Unlike traditional QA roles focused solely on deterministic software behavior, this role is accountable for testing AI-generated outputs, multi-agent workflows, execution traceability, and lifecycle integrity across complex, evolving work states. This role ensures that our AI agents produce accurate, explainable, compliant, and reproducible results while adhering to platform invariants such as attribution, auditability, promotion paths, and lifecycle correctness. The AI QA Analyst will partner closely with Product and Business Analysis teams to embed quality signals directly into our execution model.

Requirements

• 4+ years of experience in Quality Assurance, Test Engineering, or Product Quality roles within complex software platforms.
• Hands-on experience testing AI-enabled, data-driven, or workflow-oriented systems.
• Proven ability to validate non-deterministic outputs where "correctness" is probabilistic rather than binary.
• Strong understanding of QA methodologies across functional, integration, regression, and UAT testing.
• Experience designing test cases for LLMs, generative AI systems, or intelligent automation.
• Ability to reason about agent behavior, execution traces, and output quality across multi-step workflows.
• Familiarity with Agile delivery environments and tools such as JIRA, Confluence, or similar.
• Exceptional attention to detail and comfort working with complex system states and abstractions.

Nice To Haves

• Experience with agentic AI frameworks, orchestration systems, or multi-agent workflows.
• Background in regulated industries such as healthcare, life sciences, or financial services.
• Exposure to LLM evaluation techniques, prompt testing, or AI output benchmarking.
• Familiarity with audit logging, compliance validation, or responsible AI practices.

Responsibilities

• Agentic AI Output Validation Design and execute QA strategies to validate AI-generated artifacts, recommendations, and decisions produced by autonomous and semi-autonomous agents.
• Evaluate outputs for accuracy, relevance, completeness, hallucination risk, and alignment with provided context modules and knowledge sources.
• Validate that agent outputs are attributable, traceable, and reproducible within our execution and memory model.
• Multi-Agent Workflow & Lifecycle Testing Test end-to-end agent workflows across the full lifecycle: Intent → Objective → Plan → Execution → Artifacts → Memory → Evolution
• Validate correct behavior across both ad-hoc requests and structured deliverables, including seamless promotion between the two without loss of context or artifacts.
• Ensure lifecycle state transitions, dependencies, and blocking logic behave deterministically and transparently.
• Platform & Execution Quality Assurance Develop test cases for plan creation, phase execution, step execution, retries, halts, and re-runs across human and AI agents.
• Validate execution logs, artifacts, decisions, and audit events for completeness and consistency.
• Test failure modes, edge cases, and recovery scenarios in agent execution paths.
• Compliance, Governance & Explainability Partner with Legal, Compliance, and Data teams to ensure QA coverage for regulated workflows, including healthcare-grade auditability and review readiness.
• Validate that AI outputs meet governance requirements for explainability, source grounding, and decision rationale.
• Ensure platform behavior aligns with responsible AI standards, including bias awareness and data usage constraints.
• Continuous Quality Improvement Contribute to the design of automated testing frameworks for agentic systems, including regression testing of AI outputs over time.
• Define and track quality metrics related to AI performance, output stability, and user trust signals.
• Provide structured feedback loops to Product and Engineering teams to improve agent behavior, prompts, and orchestration logic.

Benefits

• Inizio Evoke provides an exciting and flexible work environment, great compensation packages and outstanding company paid benefits including medical, dental, 401(k), tuition reimbursement and flexible time off.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed