AI Tester

OmegaHires•Los Angeles, CA

1d•Hybrid

About The Position

We are seeking a detail-oriented AI Tester to ensure AI-driven products perform accurately, safely, and reliably across real-world scenarios. This role will partner closely with Product, Engineering, and Subject Matter Experts (SMEs) to validate model behavior, assess data quality, and identify trust-related risks throughout the AI lifecycle.

Requirements

Strong experience in QA/test planning, exploratory testing, and UAT
Solid understanding of AI/ML fundamentals and evaluation metrics
Ability to analyze data and reason about model behavior and outputs
Strong domain understanding and user-centric mindset
Excellent documentation and communication skills

Nice To Haves

Experience testing GenAI prompts and responses
Experience working with AI/ML or data-driven products
Certifications such as ISTQB, AI/Data certifications, or Responsible AI training

Responsibilities

Evaluate AI/ML and GenAI outputs for accuracy, consistency, edge cases, and failure modes
Validate training and inference data quality, identifying risks such as bias and data leakage
Design, implement, and execute model evaluation and regression testing strategies
Maintain and expand Golden Datasets for regression testing across model updates, prompt changes, and releases
Test AI behavior within end-to-end workflows and system integrations
Support Responsible AI initiatives, including fairness, explainability, and compliance validation
Monitor model performance post-release, identifying drift, regressions, and user feedback trends
Develop and enhance test automation coverage for AI systems
Track and report quality metrics such as accuracy, precision, and recall, highlighting gaps, errors, and hallucinations
Clearly communicate testing outcomes, risks, and recommendations to stakeholders