QA/Test Engineer

Weekday AI

6d • $60 - $90 • Remote

About The Position

This role is for one of our clients. Join a pioneering AI initiative focused on building the next generation of evaluation benchmarks for frontier AI models. We are seeking experienced QA and Test Engineers to ensure every benchmark is reliable, reproducible, and accurately measures real AI capabilities. In this role, you will review complex, multi-step evaluation tasks, validate their correctness, identify edge cases, and strengthen testing methodologies before benchmarks are deployed. You'll collaborate closely with AI researchers and task authors to improve evaluation quality, eliminate ambiguity, and ensure benchmark integrity. This is a fully remote, full-time engagement requiring approximately 35 hours per week.

Requirements

• Master's degree, PhD, or equivalent practical experience in a STEM discipline involving software engineering, research, or advanced technical problem solving.
• Minimum 1 year of professional experience in Quality Assurance, Test Engineering, Software Engineering, Research Engineering, or a related technical field with strong quality ownership.
• Proven experience designing test cases, validating complex software systems, and debugging end-to-end workflows.
• Strong proficiency in Python and Git, with the ability to troubleshoot unfamiliar codebases and technical environments.
• Excellent analytical thinking, problem-solving skills, and exceptional attention to detail.
• Experience documenting bugs, test strategies, and technical findings with clear written communication.
• Ability to work independently while managing multiple complex tasks with minimal supervision.
• Ability to commit approximately 35 hours per week on a consistent basis.

Nice To Haves

• Experience evaluating AI systems, machine learning models, or AI-generated outputs is preferred.
• Experience with AI evaluation, benchmark development, or quality assurance for machine learning systems.
• Background in automation testing, validation frameworks, or software quality engineering.
• Familiarity with large language models, AI agent workflows, or evaluation pipelines.
• Experience creating repeatable QA processes for research or engineering projects.

Responsibilities

• Design comprehensive test cases that validate evaluation tasks, including complex edge cases and unexpected scenarios.
• Review benchmark tasks and reference solutions to identify ambiguity, inconsistencies, missing requirements, and grading gaps.
• Debug task environments and Python-based evaluation scripts to ensure reliable execution and accurate results.
• Develop repeatable quality assurance processes, validation checklists, and testing frameworks for benchmark creation.
• Identify potential shortcuts, exploits, or weaknesses that could compromise evaluation accuracy or benchmark integrity.
• Collaborate with AI researchers, engineers, and task authors to improve task quality, reproducibility, and technical rigor.

Benefits

• Play a key role in improving the quality and reliability of next-generation AI evaluation benchmarks.
• Collaborate with leading AI researchers on cutting-edge evaluation methodologies.
• Help ensure AI systems are tested against rigorous, real-world scenarios.
• Apply your software testing and quality engineering expertise to advance AI reliability.
• Enjoy the flexibility of a fully remote engagement while contributing to impactful AI research.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior