Product Analyst

Dynamo AI•San Francisco, CA

About The Position

Our PMs each own a set of business problems, such as getting AI agents approved for production, reducing the effort legal and compliance teams spend on guardrails, or lowering the cost of running guardrails at scale. Almost every one of these problems depends on evidence: whether an approach works, how well, and at what cost. Product Analysts produce that evidence. Working with the PM who owns a problem, you take a specific question within it and own the answer. You scope the question, design the experiment or prototype, define how success is measured, run the analysis, and come back with a recommendation the PM can act on. When building is faster than waiting, that includes standing up a quick tool with AI to test an idea. You may support several PMs as priorities shift. Training and infrastructure code is handled by our engineering and research teams. Your contribution is experimental design, prototyping and analytical rigor.

Requirements

A degree with strong quantitative or analytical content.
A clear, demonstrated framework for thinking through problems.
Good instincts for data, ML concepts, and what makes an experiment or metric trustworthy.
Python and pandas for working with data.
Interest in AI security or safety, and in adversarial thinking.
Attention to detail and clear writing, including the ability to turn results into a recommendation a PM can act on.

Nice To Haves

ML coursework, or coursework or projects in statistics or NLP.
Reading or working proficiency in Japanese, Chinese or a European language.
SQL.
A habit of building small tools or scripts, including with AI assistants, to answer your own questions.
Exposure to LLMs, AI agents, security tooling, experimental design or annotation work.

Responsibilities

Design and curate the data used to train and accept custom guardrails, keep labeling consistent as datasets grow, and measure whether updates made from production errors improve performance without regressions.
Benchmark detection against new attack techniques, languages and modalities, and scope what closing each gap would take.
Assess whether synthetic evaluation and red-teaming data is faithful and diverse enough for customers' approvers to trust, and research new ways to evaluate AI applications and agents.
Compare guardrail and orchestration configurations on accuracy, latency and inference cost.
Prototype ways to aggregate and prioritize guardrail alerts, and measure their effect on the workload of security analysts.
Build and test prototypes of new workflows, such as how legal and compliance reviewers contribute to guardrail policies or how customers onboard an application on their own, and measure the time and effort they take.
Build small custom tools with AI to validate a hypothesis or unblock the team.
Measure performance with standard classification metrics (FNR, FPR, precision, recall) and report what is working and what is not.