Senior Software Engineer, Agent Runtime & Customer Workflows
About The Position
Klaviyo is hiring a Senior Software Engineer for its Customer Agent team, which builds Klaviyo’s AI-native conversational platform. The role focuses on designing and building scalable backend systems and AI-powered product experiences that enable agents to reason, retrieve context, use tools, and take reliable actions on behalf of customers. The engineer will independently own meaningful technical problems, make pragmatic tradeoffs, and help evolve system architecture and quality. The position involves building reliable backend services and APIs, production agentic systems, and asynchronous distributed workflows for high-volume AI workloads. It also addresses applied AI challenges such as retrieval-augmented generation, grounding, model selection, hallucination mitigation, and choosing deterministic or hybrid approaches when appropriate. The engineer will develop evaluation strategies, improve production reliability, participate in on-call, collaborate directly with customers and cross-functional partners, and use customer behavior and system metrics to iterate on shipped experiences. The role may require up to 10% travel, coordinated in advance.
Requirements
- 5–7 years of professional software engineering experience, with a strong foundation in backend systems and distributed applications.
- Hands-on experience building and deploying generative AI or agentic AI applications into production for real users.
- Proficiency in Python and modern backend frameworks such as FastAPI or Django.
- Experience with asynchronous processing and distributed task or event systems such as Celery, Kafka, SQS, RabbitMQ, or Redis.
- Strong knowledge of databases, data modeling, APIs, and persistence patterns used in production systems.
- Comfort working in cloud-native environments with technologies such as AWS, containers, Kubernetes, infrastructure automation, or CI/CD systems.
- Ability to reason about tradeoffs among quality, latency, cost, reliability, and implementation complexity.
- Ability to operate independently in ambiguous environments, break down loosely defined problems, and make pragmatic decisions about what to build first.
- Experience with and genuine interest in working directly with external customers to understand problems and improve products.
- Care for shipping useful, reliable products, with curiosity and sound engineering judgment.
Nice To Haves
- Experience training, fine-tuning, distilling, or otherwise adapting machine learning or language models for production use.
- Experience with reinforcement learning or feedback-driven optimization.
- Experience building evaluation infrastructure such as LLM-as-judge systems, evaluator calibration frameworks, benchmark suites, or AI quality platforms.
- Experience operating AI systems at substantial scale, including optimizing inference cost, latency, or throughput.
Responsibilities
- Design and build reliable, scalable backend systems and APIs for AI-driven customer experiences.
- Build and evolve production agentic systems involving tool use, orchestration, context management, retrieval, structured outputs, and multi-step workflows.
- Address applied AI problems involving retrieval and RAG, grounding, model selection, hallucination mitigation, and deterministic or hybrid approaches.
- Develop evaluation strategies using representative datasets, automated and human evaluation, regression testing, qualitative error analysis, and production signals.
- Improve production reliability through guardrails, fallbacks, retries, observability, monitoring, and thoughtful failure handling.
- Build asynchronous and distributed processing workflows for high-volume AI workloads.
- Participate in an on-call rotation and diagnose, mitigate, and learn from production incidents.
- Work directly with external customers and cross-functional partners to understand workflows and pain points and translate feedback into product and technical improvements.
- Monitor shipped experiences and use customer behavior, quality metrics, and system performance to guide iteration.
Benefits
- Comprehensive health, welfare, and wellbeing benefits, based on eligibility.
- The total compensation package may include participation in the annual cash bonus plan, equity, and sign-on payments.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
