Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift•Lebanon, PA

18h

About The Position

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a project-based opportunity, not permanent employment. We are building a dataset to evaluate AI coding agents, assessing their performance on real-world developer tasks. The role involves creating challenging tasks and evaluation criteria within realistic simulated environments, including building virtual companies with codebases, infrastructure, and development history. You will design tasks from intermediate environment states, craft prompts, define success criteria, and write tests to verify AI agent solutions. This includes iterating on tasks and tests based on QA feedback to ensure fair and robust evaluation. This role is not data labeling, prompt engineering, or writing code from scratch, as the AI agent will write most of the code.

Requirements

8+ years in software development
Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
Experience writing tests (functional, integration)
English proficiency - B2+

Nice To Haves

Deep understanding of where AI models fail in coding tasks.
Ability to create tasks that genuinely challenge the best AI coding models.
Skill in writing tests that accept all correct solutions and reject incorrect ones for tasks with many valid solutions.

Responsibilities

Build realistic developer environments (virtual company with codebase, infrastructure, context like tickets, docs, conversations, and development history).
Design tasks from intermediate states of these environments, crafting prompts, defining 'solved' criteria, and ensuring solvability by an AI agent.
Write tests that verify agent solutions, accepting all valid approaches and rejecting incorrect ones.
Iterate on tasks and tests based on QA feedback, reviewing agent solutions, analyzing failures, and refining evaluations.