Product Analyst

Dynamo AI•San Francisco, CA

About The Position

Our Product Managers (PMs) are responsible for a range of business challenges, including obtaining approval for AI agents in production, reducing the workload for legal and compliance teams regarding guardrails, and optimizing the cost-effectiveness of running guardrails at scale. A critical component of addressing these challenges is the generation of evidence to support decision-making. Product Analysts are tasked with producing this evidence. In collaboration with the PM overseeing a specific problem, you will take ownership of a particular question within that problem space. Your responsibilities will include scoping the question, designing experiments or prototypes, defining success metrics, conducting analyses, and presenting actionable recommendations to the PM. This may also involve developing quick tools using AI to test hypotheses when rapid development is more efficient than waiting for standard processes. You might support multiple PMs as priorities evolve. While engineering and research teams handle training and infrastructure code, your primary contributions will be in experimental design, prototyping, and analytical rigor.

Requirements

A degree with strong quantitative or analytical content (e.g., physics, information or data theory, business).
Solid data analysis experience.
A clear, demonstrated framework for thinking through problems.
Good instincts for data, ML concepts, and what makes an experiment or metric trustworthy.
Proficiency in Python and pandas for working with data.
Interest in AI security or safety, and in adversarial thinking.
Attention to detail.
Clear writing skills, including the ability to turn results into actionable recommendations.

Nice To Haves

ML coursework, or coursework or projects in statistics or NLP.
Reading or working proficiency in Japanese, Chinese, or a European language.
SQL.
A habit of building small tools or scripts, including with AI assistants, to answer your own questions.
Exposure to LLMs, AI agents, security tooling, experimental design, or annotation work.

Responsibilities

Scope specific questions within broader product problems.
Design experiments or prototypes to gather evidence.
Define success metrics for experiments and analyses.
Conduct analyses and interpret results.
Present actionable recommendations to Product Managers.
Develop quick tools with AI to test hypotheses when necessary.
Design and curate data for training and accepting custom guardrails.
Ensure consistency in labeling as datasets grow.
Measure the performance of guardrail updates.
Benchmark detection against new attack techniques, languages, and modalities.
Assess the faithfulness and diversity of synthetic evaluation and red-teaming data.
Research new methods for evaluating AI applications and agents.
Compare guardrail and orchestration configurations on accuracy, latency, and inference cost.
Prototype methods for aggregating and prioritizing guardrail alerts.
Measure the effect of alert prioritization on security analyst workload.
Build and test prototypes of new workflows (e.g., legal/compliance review, customer onboarding).
Measure the time and effort required for new workflows.
Build small custom tools with AI to validate hypotheses or unblock teams.
Measure performance using standard classification metrics (FNR, FPR, precision, recall).
Report on what is working and what is not.