AI/ML Product Manager

Disco•New York, NY

2d•$190,000 - $220,000•Hybrid

About The Position

Disco is a commerce media company run like an AI company. We brought our bidding and ad-serving stack in house. We built an agentic development pipeline where every pull request, every message, and every review flows through our own harness. Our CEO ships models into production through AI agents. That was the first act. The second act is the part nobody owns yet: the layer that ties all of it together. The model orchestration, the harnesses, the eval loops, the agent graphs, and the routing logic that decides which model answers which problem. Today that layer is a collection of sub-agents pointed at tasks. We want it rethought, productized, and owned. We're hiring an AI/ML Product Manager to own Disco's AI platform end to end. You're the person who makes sure we're model-agnostic and harness-agnostic, so when a vendor jacks up prices or a better model ships tomorrow, we swap in a day instead of re-platforming. You own the eval loops that tell us what's working, the harnesses that make AI safe to push to production, and the agentic pipeline that lets a team of 25 move like a team of 200. This is a product role with real technical depth. You won't just write specs and hand them to engineers. You'll understand the orchestration layer well enough to redesign it, evaluate models and harnesses against each other, and dive into the data and the code to see where tokens are being burned, where an eval is lying, or why a workflow broke. You're the closest thing Disco has to an owner of the AI operating system.

Requirements

Shipped real AI products or platforms into production, and can point to the eval loops, harnesses, or orchestration built around them.
Thinks in systems: routing, evaluation, feedback loops, cost. Builds the harness that makes AI safe and the eval that proves it works.
Technically credible without being a full-time engineer. Can read code, query data, and reason about model tradeoffs, latency, and cost.
Model-agnostic by instinct. Cares about outcomes, not loyalty to a vendor.
Strong product judgment and a bias toward shipping. Can take a fuzzy mandate like 'own the AI platform' and turn it into a roadmap, then execute it.
Comfortable with a level of ambiguity most PMs never see. Will build the thing they want to use.
Communicates clearly across a small, fast team. Can explain a model routing decision to the CEO and a harness change to an engineer in the same week.
Can dive into the data and the code to understand what's actually happening for any given workflow, eval, or cost anomaly.

Responsibilities

Own the layer that routes every task to the right model: reasoning to the frontier models, volume to the cheap ones, image and creative work to the models that are actually good at it. This is bespoke routing, not one model for everything.
Keep Disco model-agnostic and harness-agnostic. When a vendor raises prices 10x or a better model drops, we swap without being beholden to any single LLM contract.
Track the total cost of every model call. Own the LLM budget like it's your P&L, because it's about to become one of the largest OpEx lines in the company.
Own the harnesses that wrap AI so it can ship to production safely: what gets reviewed, what gets verified, what a non-ML user can and cannot push to traffic.
Build and own the eval loops that tell us what's actually working. A model, an agent, a workflow, a cron — if it runs on AI, there's an eval that says whether it's good, and you own that eval.
Design eval graphs that catch regressions before they cost money, and the guardrails that keep an enthusiastic experiment from becoming a production incident.
Own the agentic development pipeline: the AWS Bedrock harness, the automated PR flow, the QA gates. Make it faster, cheaper, and more reliable.
Productize the internal AI tooling. Cron generation, workflow automation, memory, context — these are scripts today. They should be products with owners, versions, and evals.
Attack the silent cost centers. Crons and background jobs that burn LLM tokens are the first target: turn expensive agent loops into cheap, deterministic code where it belongs.
Make every team agentic. When a non-technical teammate wants to build, evaluate, or automate something with AI, your harnesses and loops are what make that safe.
Own the knowledge of which models are best at what, and keep it current. The model landscape changes monthly; the team shouldn't have to track it.
Set the standard for where AI output is trusted and where it gets verified before it ships.