Senior AI Application Engineer

Klaviyo•Boston, MA

6d•$132,000 - $198,000•Hybrid

About The Position

At Klaviyo, the ARIA (Applications, Resources, Intelligence, and Automations) team is building the AI systems that transform how our employees work. We are looking for a Senior IT App Engineer to design, build, and scale production-grade LLM integrations and AI-powered automation tools from the ground up. This role sits at the intersection of applied AI research and full-stack software engineering. You will own everything from LLM integration architecture and RAG system design to CI/CD pipeline management and production observability. If you want to build AI that real people use at scale every day, this is the role for you.

Requirements

3+ years of software engineering experience, with at least 2 years focused on building AI, ML, or LLM-powered systems in production.
Deep expertise with large language models such as OpenAI, Anthropic, Mistral, or Llama, and experience integrating them into production applications via API.
Strong full-stack engineering skills across Python (FastAPI, Flask, or Django) and TypeScript or JavaScript, with experience in modern frontend frameworks.
Proven experience designing and building CI/CD pipelines for software or ML systems using tools such as GitHub Actions, Jenkins, or CircleCI.
Experience with RAG systems, vector databases (Pinecone, Weaviate, pgvector), and embedding models.
Proficiency with cloud infrastructure (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code tools such as Terraform or Pulumi.
Experience with prompt engineering, model evaluation frameworks, and LLM observability tools such as LangSmith or Weights and Biases.

Responsibilities

Design and build LLM-powered applications: Architect and develop production-grade systems leveraging large language models (GPT-4, Claude, Mistral, Llama) for internal automation, knowledge retrieval, and intelligent workflow assistance.
Build and maintain AI integration pipelines: Design robust, scalable API integrations between AI models and internal systems such as HRIS, ticketing, CRM, and knowledge bases, ensuring reliability, low latency, and high availability.
Develop full-stack AI features end-to-end: Own the complete development of AI features across the stack from backend model orchestration and API layers to frontend interfaces that make AI accessible to non-technical employees.
Architect and manage CI/CD pipelines for AI systems: Build and maintain automated deployment pipelines for AI models and services, including model evaluation frameworks, A/B testing infrastructure, and safe rollout strategies.
Implement RAG and retrieval systems: Design and build retrieval-augmented generation systems grounded in Klaviyo internal knowledge, ensuring accuracy, relevance, and up-to-date responses at scale.
Evaluate and fine-tune AI models: Lead benchmarking, evaluation, and fine-tuning of foundation models for Klaviyo use cases, including prompt engineering and parameter-efficient fine-tuning techniques.
Establish AI engineering best practices: Define coding standards, testing frameworks, and architectural patterns for AI development at Klaviyo, enabling the team to ship reliable AI features consistently.
Build AI observability and monitoring tooling: Instrument AI systems with latency monitoring, output quality evaluation, hallucination detection, cost dashboards, and production alerting.
Collaborate with Product and Data Science: Work closely with PMs, data scientists, and ML engineers to translate requirements into technical implementations, maintaining a tight feedback loop between experimentation and production.
Drive security and compliance in AI systems: Ensure AI integrations follow data privacy best practices, implement guardrails for sensitive data handling, and maintain compliance with Klaviyo security and governance standards.