Senior AI Product Engineer

Progress Partners

13d • Remote

About The Position

We're looking for a Senior AI Product Engineer to own the continuous improvement of our customer-facing AI products, making them more accurate, reliable, cost-efficient, and genuinely useful to the people who rely on them every day. This isn't a prompt-writing job, it's about building the structured loop of evaluation, experimentation, and analytics that turns AI from a feature into a measurable product advantage. You'll work at the intersection of Product, Growth, and Engineering, with real influence over how AI behavior gets measured, improved, and shipped.

Requirements

• 5+ years in software engineering, AI product engineering, ML engineering, or a related technical role, plus 2+ years hands-on with LLMs, prompt engineering, or applied AI systems.
• Proven experience improving AI features post-launch through analytics, experimentation, and structured evaluation, not just initial prompt design.
• Strong grasp of product analytics, funnel metrics, and how to connect AI behavior to business outcomes.
• Ability to diagnose whether an AI quality issue stems from prompts, context, model limitations, UX, or data, and to fix the right layer.
• Strong written communication and the ability to work autonomously in a remote, async environment.
• Comfort working closely with Product, Growth, Design, Engineering, QA, and Analytics.
• Comfort with ownership, transparency, and a results-driven mindset.

Nice To Haves

• Experience with A/B testing frameworks, AI observability tools, or custom evaluation pipelines.
• Hands-on work with OpenAI, Anthropic, Gemini, AWS Bedrock, or similar LLM providers.
• Familiarity with RAG, embeddings, semantic search, or tool-calling architectures.
• Background in chatbot, assistant, content generation, or recommendation products.

Responsibilities

• Own the end-to-end quality of our AI-powered product experiences, improving response usefulness, consistency, and reliability in ways users actually notice.
• Build evaluation frameworks and repeatable test sets so we can measure AI quality objectively, not anecdotally.
• Design and run experiments on prompts, context strategies, and model settings, connecting every change to activation, conversion, retention, or satisfaction metrics.
• Monitor and reduce AI-related costs (tokens, latency, model choice, retrieval) without compromising the user experience.
• Translate AI failure patterns, hallucinations, inconsistent outputs, weak reasoning, into clear, prioritized fixes for Product and Engineering.
• Help Product, Growth, and Engineering teams understand where AI changes create real user and business impact.

Benefits

• Remote-first: work from anywhere in the world, no relocation required.
• Flexible hours: we measure outcomes, not time at a desk.
• Unlimited PTO: take the time you need to rest, recharge, and do your best work.
• Learning & development budget: courses, books, conferences, certifications, whatever helps you grow.
• Performance-based compensation: your impact directly shapes your pay and career trajectory.
• A small, senior, collaborative team where your work is visible and your decisions matter.
• The chance to build products used by millions of people across 160+ countries.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed