Staff Product Manager - AI Evals
About The Position
Twenty Technologies is seeking a Staff Product Manager to help make its agentic AI systems measurable, reliable, and continuously improvable. The role partners with AI and engineering teams to shape evaluation capabilities, including defining agent performance and product quality and turning evaluation results into useful workflows and product decisions. The position is suited to a product-minded, technically curious operator with experience building or managing agentic systems, either through direct evaluation work or by building systems that enable teams to evaluate and improve agents. Cyber experience is a plus but is not required. This is an onsite, full-time role in a mission-driven company focused on offensive cyber operations for the U.S. and its allies. U.S. citizenship is required; no active clearance is required.
Requirements
- Strong product experience with AI or agentic systems, including experience building or managing agentic systems through product work.
- Experience directly building evaluations for agents or building the evaluation product, tooling, or system used to assess them.
- Ability to translate technical evaluation needs into product requirements, priorities, and workflows in partnership with engineering and AI teams.
- Ability to define or operationalize meaningful measures of agent performance and product quality.
- Strong communication, collaboration, and product judgment in an evolving, technically complex environment.
- U.S. citizenship is required.
Nice To Haves
- Experience in cybersecurity, offensive cyber, government, defense, intelligence, or another mission-driven environment.
- Familiarity with evaluation approaches such as benchmarks, human or automated evaluation, regression testing, simulation, or analysis of agent traces.
- Experience working with products that combine autonomous systems with meaningful human judgment and oversight.
Responsibilities
- Own or support the product roadmap for agentic AI evaluations, translating product-area goals into clear priorities and outcomes.
- Partner with engineering and AI teams to understand evaluation needs, technical constraints, and user workflows.
- Define how agent performance and product quality should be assessed, including useful measures of task success, reliability, and quality.
- Translate technical evaluation needs into product requirements, workflows, and acceptance criteria.
- Help operationalize evaluation workflows and success metrics so teams can consistently assess agent behavior and product changes.
- Use evaluation results and feedback to identify gaps, inform product decisions, and improve the agent experience over time.
Benefits
- Medical, dental, and vision plan options.
- Life and AD&D coverage options.
- Disability coverage options.
- Paid parental leave for eligible full-time employees: 12 weeks for birthing parents, 4 weeks for non-birthing parents, and 6 weeks for adoptive, foster, or intended parents through surrogacy.
- Paid holidays and flexible PTO.
- 401(k) with pre-tax and Roth options.
- HSA/FSA options, including dependent care FSA.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
