About The Position
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. This opportunity involves building a dataset to evaluate AI coding agents by assessing how well a model handles real-world developer tasks. You will create challenging tasks and evaluation criteria within realistic simulated environments.
Requirements
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Nice To Haves
- Experience in QA or Security
Responsibilities
- Build realistic developer environments, including a virtual company with a codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history.
- Design tasks from intermediate states of these environments, crafting the prompt, defining what "solved" means, and ensuring the task is solvable by an AI agent.
- Write tests that verify agent solutions, accepting all valid approaches and rejecting incorrect ones.
- Iterate on tasks and tests based on QA feedback, reviewing agent solutions, analyzing failures, and refining until the evaluation is fair and robust.
Benefits
- Project-based opportunities
- Up to $50/hr equivalent compensation
- Tasks estimated at ~20 hours each
- Set your own schedule
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Part-time
Career Level
Senior
Education Level
No Education Listed
