Staff Technical Product Manager, Evals

Hercules•San Francisco, CA

1d•Hybrid

About The Position

Hercules is seeking a senior- or staff-level product manager to own evaluations for its AI coding agent. The role’s sole goal is to make the agent cheaper, faster, and more accurate. The product manager will own Hercules Bench, the offline evaluation set, and Hercules’ online evaluations, using the company’s user, repository, and chat data to inform evaluation creation while anonymizing data to protect customer privacy. The role combines technical product management, evaluation design, product and design judgment, and knowledge of AI context and harness engineering. The position works with an engineering counterpart on complex implementation. Hercules’ core stack includes TypeScript, React, AI SDK, and an evaluation platform such as LangSmith. The broader technology stack includes Postgres, Redis, ClickHouse, Tailwind, infrastructure-as-code tools, Docker, OpenTelemetry, Hono, Drizzle ORM, Temporal, Cloudflare, AWS, and models from OpenAI, Anthropic, and Google. Hercules emphasizes shipping quickly, iterating, striving for excellence, attention to detail, simple and self-serve outputs, and working hard to achieve outcomes. The founding team works in an office in San Francisco, and the company strongly prefers in-office work. Remote work is open to exceptional candidates who can work in US time zones and travel to the office regularly. Compensation is $200,000–$350,000 plus equity, variable based on seniority, equity, and location.

Requirements

Strong software fundamentals and the ability to code.
A decent understanding of system design, architecture, data modeling, and API design.
Explicit experience creating and/or managing technical evaluations.
Ability to work across a variety of evaluation methodologies and deeply understand optimal reward modeling.
Strong intuition for customer goals and novel solutions that help customers achieve them.
A track record of building simple, delightful user experiences that help customers achieve their goals.
Strong experience with current AI context and harness engineering practices, including summarization and compression, prompting, prompt caching, subagents, memory, context management, tools, asynchronous and deferred tools, and skills.
Experience wrestling with AI context and harness engineering challenges at scale and strong opinions about what works.
Ability to use frontier AI coding tools to substantially increase workflow productivity.

Responsibilities

Own Hercules Bench, the offline evaluation set for Hercules’ AI coding agent.
Own Hercules’ online evaluations.
Make the AI coding agent cheaper, faster, and more accurate.
Use data from Hercules’ users, code repositories, and chat threads to guide evaluation creation.
Anonymize data to protect customer privacy.
Work with an engineering counterpart on complex implementation.