Sr. AI Engineer

Vertafore • Denver, CO

5d • Onsite

About The Position

As a Senior AI Engineer, you will design, build, and operate AI systems for Vertafore’s next generation of products for agencies, MGAs, and carriers. You will build production-grade agents that reason through complex, multi-step insurance workflows and ensure those systems are reliable, measurable, and safe in a highly regulated industry. You will work in small, autonomous pods to build end-to-end products and directly contribute to implementing and launching critical features. The role expects fluidity across the technology stack, hands-on engineering, and accountability for the quality, security, and correctness of shipped work. You will use current agentic coding tools to work efficiently.

Requirements

• Bachelor’s degree in Computer Science, Engineering, Physics, Mathematics, or a related field, or equivalent practical experience.
• At least 7 years of professional software engineering experience, including significant experience building backend services and systems in Python.
• At least 2 years building and shipping LLM-powered applications or agentic systems to production.
• Hands-on production experience with LangGraph or comparable agent orchestration frameworks such as LangChain, LlamaIndex, or custom orchestration.
• At least 5 years designing and building RESTful APIs.
• Demonstrated experience designing and running evaluations for ML or LLM systems and using results to drive measurable quality improvements.
• Experience with cloud platforms, Docker, and CI/CD pipelines.
• Expert-level Python skills, including idiomatic, typed, tested, production-quality code and asynchronous programming.
• Solid understanding of ML fundamentals, including model training and generalization, overfitting, bias and variance, embeddings, classification and ranking metrics, and statistical significance.
• Deep understanding of practical LLM behavior, including failure modes, non-determinism, context limits, and strategies for reliable outputs.
• Rigorous, experimental mindset: form hypotheses, measure outcomes, and use data rather than intuition about model output to guide decisions.
• Must be legally authorized to work in the United States.

Nice To Haves

• Experience with FastAPI or similar frameworks.
• Experience with LLM observability and evaluation tools such as LangSmith, Langfuse, Braintrust, or similar.
• Experience with fine-tuning, embedding model selection, or classical ML pipelines.
• Experience in insurance, financial services, healthcare, or another regulated industry.
• Experience with Model Context Protocol (MCP) or building tool integrations for agents.
• Open-source contributions or public writing on AI engineering.

Responsibilities

• Design, build, and deploy complex reasoning agents and multi-agent systems for production workloads using LangGraph and related Python orchestration frameworks.
• Architect agent workflows, including state management, tool use, memory, planning, human-in-the-loop checkpoints, error recovery, and fallback behavior.
• Design and build clean, documented, versioned APIs that expose AI capabilities to Vertafore products and external partners.
• Define and own evaluation strategies for AI systems, including evaluation datasets, programmatic, model-based, and human graders, offline and online evaluations, and improvements based on results.
• Set up regression testing for prompts, models, and agent behavior so changes are supported by measurable confidence.
• Instrument AI systems for production observability, including tracing, latency, cost, token usage, failure modes, and quality metrics.
• Build retrieval and context pipelines, including RAG, document processing, and structured extraction, over insurance data such as policies, submissions, and compliance documents.
• Troubleshoot and resolve production issues across the AI stack, from model behavior to infrastructure.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior