Software Engineer, AI Agent & LLM
About The Position
We are looking for a Senior AI Agent & LLM Engineer who combines strong software engineering capabilities with a deep focus on AI quality. You will help build and improve the AI systems behind Otter’s conversational knowledge engine and AI Chat, spanning user-facing agent experiences, evaluation systems, and the shared platform required to operate them reliably at scale. This is a hands-on role for someone who can move from an ambiguous product problem to a working production solution, define how quality should be measured, and drive improvements across services, models, and user experience.
Requirements
- Has 3+ years of AI Agent engineering, machine learning engineering, or related experience.
- Brings strong backend or distributed-systems engineering skills.
- Has hands-on experience building and shipping AI-agent or LLM-powered products.
- Has a strong focus on AI quality and experience evaluating nondeterministic systems.
- Can use data, traces, logs, and qualitative examples to identify and resolve complex failures.
- Works effectively across services, technical domains, and organizational boundaries.
- Combines strong product judgment with rigorous engineering and evaluation practices.
- Operates with high agency, strong ownership, and a bias toward action.
- Can take ambiguous problems from initial exploration through production launch and continuous improvement.
Nice To Haves
- Demonstrated ability to use coding agents effectively while rigorously reviewing and controlling the quality of their output is a plus.
Responsibilities
- Build AI-agent experiences for Otter AI Chat that help users reason over conversations, retrieve knowledge, and complete complex tasks.
- Advance Otter’s conversational knowledge engine through better knowledge extraction, annotation, and indexing.
- Develop robust evaluation datasets, automated graders, regression tests, and release gates for AI quality.
- Diagnose failures across models, prompts, retrieval, tools, data pipelines, backend services, and product workflows.
- Build shared agent infrastructure for orchestration, tracing, debugging, retries, sandboxed execution, and observability.
- Improve task completion, correctness, groundedness, reliability, latency, and cost.
- Turn production traces, customer feedback, and usage signals into measurable product and model improvements.
- Collaborate across AI, product, infrastructure, data, security, and application teams to deliver end-to-end capabilities.
Benefits
- We provide reasonable accommodations for qualified applicants throughout the hiring process.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
