Applied AI Engineer, Video Agent

HeyGen•Los Angeles, CA

5d•$180,000 - $240,000

About The Position

The Video Agent team at HeyGen is responsible for developing an AI agent that transforms prompts, documents, or assets into finished videos. This agent handles multiple steps including planning a storyboard, writing scripts, selecting avatars and voices, utilizing various tools, and integrating with the rendering pipeline. The team owns the agent end-to-end, encompassing the prompts and tools it uses, the backend services, and the user-facing product screens for interaction and editing. This role offers the unique opportunity to work directly with in-house trained models and their researchers, rather than just calling external APIs. The Applied AI Engineer will focus on improving the agent's ability to accurately fulfill user requests, validated by data. Responsibilities include redesigning agent tools, building evaluation frameworks, running A/B tests, and developing new editing experiences. The role involves owning features from conception to launch and beyond, and determining the division between model-based and deterministic code solutions.

Requirements

Experience building and shipping an AI agent that real people use, involving model tool-calling, multi-step planning, and acting on results.
Experience building evals or test sets for model behavior and ability to distinguish real improvements from noise.
2 or more years of professional full-stack engineering experience.
Fluency in TypeScript and React for frontend development.
Fluency in Python for backend development.
Strong product sense, including using what you build and caring about user experience.
Demonstrated ownership, ability to identify and fix the most important problems without waiting for a spec.

Nice To Haves

Experience fine-tuning models.
Experience with code-to-video models.
Experience with video agent models in general.

Responsibilities

Build the agent by designing the tools, prompts, and context it works with.
Improve the agent's planning capabilities, error recovery, and handling of long, multi-turn conversations.
Measure quality by building evals and offline replays of real sessions.
Run experiments behind feature flags and analyze results honestly.
Watch real user sessions and agent traces to identify issues.
Ship user-facing features using TypeScript and React.
Build backend services and long-running workflows using Python and Go.
Partner with product managers, designers, and the research and rendering teams.