AI Quality Analyst
Shyft6•New York, NY
•Remote
About The Position
We are seeking an AI Quality Analyst to support testing and validation efforts for AI-driven applications, data platforms, and client-facing solutions. This role will focus on ensuring the accuracy, reliability, and consistency of AI outputs, as well as validating data pipelines and system integrations across AWS, Azure, Tableau, Power BI, and DealCloud CRM. This is an ideal role for someone with a foundation in QA or data validation who is looking to grow into AI/ML testing and validation.
Requirements
- 2–5 years of experience in QA, data analysis, or testing roles
- Strong attention to detail with experience in data validation and quality assurance
- Proficiency in SQL for querying and validating data
- Experience with API testing tools (e.g., Postman or similar)
- Familiarity with cloud-based environments (AWS and/or Azure)
- Experience validating outputs in BI tools such as Tableau or Power BI
- Strong analytical and problem-solving skills
Nice To Haves
- Highly detail-oriented with a strong focus on data accuracy
- Curious mindset with interest in how AI systems behave and improve over time
- Strong collaboration skills across technical and business teams
- Ability to work in a fast-paced, evolving AI environment
Responsibilities
- Execute test cases to validate AI/ML model outputs, including accuracy, consistency, and expected behavior
- Perform data validation testing across ETL/ELT pipelines and integrated systems
- Conduct functional and regression testing for AI-driven features and applications
- Validate reporting outputs in Tableau and Power BI against underlying data sources
- Support testing of API integrations and data flows between systems, including CRM platforms like DealCloud
- Identify, document, and track defects, inconsistencies, and data anomalies
- Assist in developing and maintaining test cases, test scripts, and validation datasets
- Collaborate with data engineers, developers, and QA teams to ensure data and model quality
- Participate in user acceptance testing (UAT) and validation of business requirements
- Monitor for issues such as data drift, inconsistencies, or unexpected AI behavior
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
Number of Employees
1-10 employees
