Software Engineer - AI Agent Evaluation (Remote)

Mindrift•New York, NY

2d•Remote

About The Position

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. This role involves creating a dataset to evaluate AI coding agents by building realistic simulated developer environments, designing challenging tasks, and writing tests to verify AI solutions. The tasks are designed to challenge advanced AI models and require careful crafting of evaluation criteria to ensure fairness and robustness. This is not a data labeling, prompt engineering, or traditional code writing role, but rather focuses on guiding and evaluating AI-generated code.

Requirements

5+ years in software development
Core stack experience: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
Experience writing tests (functional, integration)
English proficiency - B2+

Responsibilities

Build realistic developer environments, including virtual companies with codebases, infrastructure, and contextual information like tickets, documentation, and conversations.
Design tasks from intermediate states of these environments, crafting prompts and defining success criteria for AI agents.
Ensure tasks are solvable by AI agents.
Write tests to verify AI agent solutions, accepting all valid approaches and rejecting incorrect ones.
Iterate on tasks and tests based on QA feedback, reviewing agent solutions, analyzing failures, and refining evaluations for fairness and robustness.