Staff Technical Product Manager, Evals
About The Position
Hercules is seeking a senior- or staff-level product manager to own evaluations for its AI coding agent. The role’s primary goal is to make the agent cheaper, faster, and more accurate. You will own Hercules Bench, the offline evaluation set, and online evaluations, using data from Hercules’ users, code repositories, and chat threads to guide evaluation creation while anonymizing data to protect customer privacy. You will partner with engineering on complex implementation and work across evaluation methodologies, reward modeling, product and design, and current AI context and harness engineering practices. Hercules has a strong preference for in-office work in San Francisco but is open to exceptional remote candidates who work in US time zones and travel to the office regularly. Compensation is $200,000–$350,000 plus equity, varying by seniority, equity, and location.
Requirements
- Senior- or staff-level product management capability.
- Technical ability to code and a solid understanding of software fundamentals, including system design, architecture, data modeling, and API design.
- Explicit experience creating and/or managing technical evaluations.
- Deep understanding of evaluation methodologies and optimal reward modeling.
- Strong product sense and intuition for customer goals and effective solutions.
- A track record of building simple, delightful UX/UI.
- Strong experience with current AI context and harness engineering practices, including summarization and compression, prompting, prompt caching, subagents, memory, context management, tools, asynchronous and deferred tools, and skills.
- Experience wrestling with AI context and harness engineering challenges at scale and informed opinions about effective approaches.
- Ability to substantially improve workflow using frontier AI coding tools.
Nice To Haves
- Experience with TypeScript, React, an agent framework such as AI SDK, or an evaluation platform such as LangSmith.
- Willingness to work in US time zones and travel regularly to the San Francisco office if working remotely.
Responsibilities
- Own Hercules Bench, the offline evaluation set.
- Own online evaluations for Hercules’ AI coding agent.
- Improve the AI coding agent’s cost, speed, and accuracy.
- Use data from Hercules’ users, code repositories, and chat threads to guide evaluation creation.
- Anonymize data to protect customer privacy.
- Partner with an engineering counterpart on complex implementation.
- Apply and work across different evaluation methodologies and reward modeling approaches.
Benefits
- Unlimited AI token budget.
- Take what you need vacation policy.
- Healthcare, vision, and dental.
- Daily lunches, dinners, and snacks in the office.
- Equity.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
