Senior Software Engineer, AI Evaluation Infrastructure

Turing

11d • $170,000 - $200,000

About The Position

Turing is hiring an experienced engineer to build and operate infrastructure that frontier AI labs use to evaluate coding agents and language models. The platform supports benchmark datasets, large-scale agent evaluation runs, and automated task-quality checks. The engineer will own features across the stack, from data models and background jobs to user-facing screens, and is expected to independently turn loosely defined requests into shipped, tested, reliable work.

The role involves building systems for evaluating agents and models, including trials in sandboxed environments, scoring, and aggregating results. It also includes improving benchmark-task quality checks through verifiers, rubric-based grading, and LLM-as-a-judge methods, and integrating new models, agents, and benchmarks in generalized ways. The engineer will help maintain performance and reliability as usage grows, review teammates’ pull requests, and uphold engineering guidelines.

The position requires substantial production backend experience, strong Python and familiarity with backend frameworks, PostgreSQL, and task queues, along with hands-on experience in LLM evaluation. Comfort making changes to a TypeScript and React frontend is also expected when needed. Compensation is $170,000–$200,000 base salary plus equity.

Requirements

• At least 5 years of experience building and operating production backend systems.
• Ownership of at least one service from design through on-call.
• Bachelor’s degree or higher in Software Engineering, Computer Science, or a related field.
• Strong Python skills and experience with Django or a similar framework, PostgreSQL, and a task queue such as Celery.
• Hands-on LLM evaluation experience beyond calling a model API, including understanding benchmark-task construction, agent-trial scoring, and aggregation and comparison of results.
• Ability to identify ambiguous task instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason.
• Sound engineering judgment, including choosing small designs that solve the problem, extending existing systems where appropriate, and explaining tradeoffs.
• Ability to independently take a loosely defined request through implementation and completion.
• Ability to quickly adopt written team conventions for naming, structure, tests, and error handling.
• Comfort working with a TypeScript and React frontend when changes require it.
• Clear written communication in pull request descriptions and design notes.

Nice To Haves

• Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar evaluation harness.
• Experience running agents in sandboxed or containerized environments, such as Docker, Kubernetes, Modal, or Daytona.
• Experience with LLM-as-a-judge or rubric-based grading, including reducing judge bias and noise.
• Familiarity with coding agents such as Claude Code, Codex, or OpenHands, and with MCP.
• Experience with distributed job scheduling, concurrency limits, or database locking at scale.
• Contributions to open-source evaluation or infrastructure projects.

Responsibilities

• Design and ship end-to-end features spanning data models, APIs, background workers, and user interfaces as needed.
• Build and improve agent and model evaluation systems, including running trials in sandboxed environments, scoring results, and aggregating large numbers of runs into trustworthy results.
• Improve benchmark-task quality checks using verifiers, rubric-based grading, and LLM-as-a-judge grading, including determining whether tasks are clear, solvable, and fairly scored.
• Integrate new models, agents, and benchmarks using generalized approaches rather than one-off special cases.
• Maintain system performance and reliability as usage grows through efficient queries, safe migrations, and concurrency that holds up under load.
• Review teammates’ pull requests against engineering guidelines and help maintain codebase consistency.
• Own work through implementation, migrations, tests, and relevant user-facing edge cases.

Benefits

• Equity
• Opportunities to contribute to leading-edge AI research and showcase work at conferences such as ICLR, ICML, and NeurIPS.
• Opportunity to collaborate with colleagues with deep AI experience from Google, Meta, Amazon, and other leading technology companies.
• Opportunity to work at the frontier of AI and bring AI innovation to enterprise challenges.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior