Senior AI Technical Product Manager - R01563914

Brillio•New York, NY

2d•$160 - $170•Remote

About The Position

This is a Technical Product Manager role. Youâll lead a team of engineers, researchers, and data scientists, and you need to be deep enough in the technology to QA AI agent outputs, evaluate RAG pipeline architectures, challenge prompt engineering decisions, review system designs, and guide the teamâs technology roadmap. Youâre not writing production code, but youâre reading it, understanding it, and making informed calls about whether the technical approach is sound. At the same time, youâre the person who builds relationships with internal business teams, understands their pain points, and translates that into the right priorities for a technical team. The difference between this role and a traditional PM is that youâre expected to be in the technical details reviewing architecture decisions, sitting in code reviews, and understanding the tradeoffs between different model choices, retrieval strategies, and deployment approaches. You own three things: what the team works on (prioritization), how well itâs done (quality), and whether it matters (stakeholder alignment). You lead planning and serve as the quality gate every piece of work passes through your technical and product review before it ships or gets shared with stakeholders. As the teamâs solutions scale, this role will also evolve into LLM cost optimization as a budget line, navigating AI governance and responsible AI practices, and guiding the technology roadmap as the AI landscape shifts.

Requirements

Youâve built things before whether earlier in your career as an engineer, data scientist, or ML practitioner, you have a hands-on technical background. You didnât start in product management; you moved into it from a technical role.
Strong understanding of LLM-based systems you know how RAG pipelines work (retrieval, embedding, re-ranking, context window management), how prompt engineering affects output quality, and how agent orchestration patterns handle multi-step workflows.
Comfortable with Python you can read Python code, understand what it does, review pull requests for logic and architecture (not just style), and write quick scripts when needed to test an idea or validate data.
Familiar with Azure cloud infrastructure you understand how AI workloads are deployed, monitored, and scaled in Azure (or equivalent cloud platforms). You can have an informed conversation about compute costs, model serving, and infrastructure architecture.
You understand the cost structure of LLM-based systems token economics, model pricing tiers, the cost implications of different prompt lengths, caching strategies, and model routing approaches. You can optimize AI spend at a technical level, not just a budget level.
You can evaluate AI outputs critically you know what a hallucination looks like, you understand why an agent might produce inconsistent results, and you can diagnose whether the issue is in the prompt, the retrieval layer, the model choice, or the data.
8+ years of experience in technical product management, engineering management, or a similar role where youâve led technical teams building AI/ML or data-driven products
Experience working with internal stakeholders who arenât technical you can translate between what a business team needs and what an engineering team should build
Experience running lean, fast-moving teams with minimal process overhead. You prefer shipping over planning.
Strong quality instinct you notice when an AI output isnât quite right, when an experiment design has gaps, or when a prototype is missing a critical edge case
Clear communicator who can present to senior leadership without hiding behind jargon or overcomplicating things
Comfort making prioritization calls with incomplete information and adjusting course as you learn more
Experience managing budgets or vendor relationships for AI/ML infrastructure, cloud services, or technical platforms
Experience with agile methodologies (Scrum, Kanban, or similar) and the judgment to adapt the process to what the team needs

Nice To Haves

Previous role as a software engineer, ML engineer, or data scientist bonus if youâve worked on NLP, search, or content generation systems
Hands-on experience building or maintaining RAG pipelines, AI agents, or LLM-based applications
Experience with AI evaluation frameworks automated scoring, human evaluation protocols, A/B testing for AI systems
Experience with certification, assessment, or education technology
Exposure to AI governance frameworks, responsible AI practices, or compliance requirements for AI-driven decision systems
Experience scaling a team or function from early-stage to a broader organizational capability
Familiarity with MLOps practices model versioning, deployment pipelines, monitoring, and cost tracking

Responsibilities

Set priorities and decide what the team works on across both AI Enablement and AI Experiments tracks
Serve as the primary liaison with internal business teams understand their workflows, gather requirements, and translate pain points into actionable technical work
Collaborate with product teams to ensure exploration work aligns with the broader product direction
Lead the teamâs operating rhythm stand-ups, demos, planning sessions, and progress readouts to leadership and stakeholders
Make resource allocation decisions across workstreams move people where theyâre needed most based on shifting priorities
Decide when to kill work that isnât delivering results youâre comfortable shutting down experiments and reprioritizing without hesitation
Communicate progress clearly to leadership and cross-functional partners what shipped, what we learned, whatâs next, and what needs their attention
Identify new internal processes where AI could meaningfully reduce effort, and build the business case to expand the teamâs scope
Guide the teamâs technology roadmap make informed decisions about model selection, infrastructure choices, build-vs-buy tradeoffs, and when to adopt new tools or frameworks. You stay current on how the AI landscape is evolving and what that means for the teamâs technical strategy.
QA all AI outputs before they move forward review agent responses for quality, accuracy, and edge case handling. You need to understand why an agent produced a given output and whether the underlying prompt, retrieval, or orchestration logic is sound.
Review and guide architecture decisions evaluate RAG pipeline designs, agent orchestration patterns, data flow architectures, and integration approaches. Youâre the person who asks âwhy did we choose this embedding model?â or âwhat happens when the retrieval layer returns irrelevant context?â and expects a rigorous answer.
Evaluate experiment designs for technical rigor when the team tests a new model or prompting approach, you review the evaluation methodology, success criteria, and whether the comparison is fair before the experiment runs, not after.
Participate in technical design reviews and code reviews not to approve every line of code, but to stay close enough to the implementation that you can catch architectural issues, scalability concerns, and technical debt before they compound.
Manage and optimize AI infrastructure spend track LLM costs, token usage patterns, and vendor contracts. You understand the cost implications of different model choices, caching strategies, and prompt designs at a technical level, not just a budget level.
Define and evolve AI governance and compliance practices as solutions handle sensitive data and higher-stakes decisions, you establish the guardrails for responsible AI use, data handling, and decision explainability.