Technology AI Evaluation Expert

Weekday AI

1d•$60 - $75•Remote

About The Position

This role is for one of our clients. Join an advanced AI research initiative focused on improving how next-generation AI systems understand professional documents, execute complex instructions, and reason through real-world technical workflows. We are seeking experienced technology professionals to design high-quality benchmark tasks that evaluate AI performance across software engineering and data science domains. In this role, you will create realistic, multi-step evaluation tasks based on technical documentation, code repositories, API references, architecture diagrams, and other workplace resources. Your work will help measure and improve the ability of AI models to interpret technical information, follow detailed instructions, and generate accurate, well-structured outputs. This is a fully remote, independent contractor opportunity with flexible working hours.

Requirements

Minimum 3 years of hands-on professional experience in one or more of the following areas: Software Engineering, Data Science, Data Analytics
Strong understanding of technical documentation, software development workflows, and engineering best practices.
Experience working with codebases, APIs, technical specifications, or system architecture documentation.
Excellent analytical thinking and problem-solving skills.
Strong written communication with the ability to create clear technical instructions and evaluation criteria.
Ability to work independently while maintaining high standards of accuracy and consistency.

Responsibilities

Create realistic, multi-step benchmark tasks based on professional technology workflows.
Develop challenges using technical specifications, architecture documents, API documentation, codebases, web research, and code execution.
Ensure each task includes a clearly defined expected output and objective evaluation criteria.
Write comprehensive ground-truth solutions and structured scoring rubrics.
Design tasks that assess reasoning, technical understanding, instruction following, and output quality.
Maintain high standards of technical accuracy, clarity, and reproducibility.
Apply real-world knowledge from software engineering, data science, or analytics to create authentic evaluation scenarios.
Collaborate with research teams to improve benchmark quality and consistency.
Continuously refine tasks based on project feedback and evolving evaluation requirements.