Software Engineer - Agentic Platform

Userpilot • Austin, TX

13h

About The Position

Userpilot is a leading product analytics and user engagement platform used by product teams at hundreds of companies to understand, segment, and activate their users. The product spans a performant JavaScript SDK that runs inside customers' web apps, a Chrome Extension for building in-app UI without code, and a React dashboard that handles complex real-time data, all backed by a distributed Elixir/Phoenix backend that sustains hundreds of thousands of concurrent WebSocket connections, high-throughput Kafka event ingestion, and real-time content delivery at scale. We move fast, we ship often, and we believe the best engineers care as much about the product they're enabling as the systems and interfaces they build.

This is an AI-deep role focused on Lia, Userpilot's agent platform, the system that turns a rich product-data model into reliable, grounded, multi-turn AI experiences. The AI is the product, not just a tool you use to build it. You'll own and elevate the agent platform: a Python service built on Microsoft Agent Framework, with hybrid retrieval over multiple tool catalogs, complex multi-step orchestration utilizing skills and sub-agents, multi-turn state and grounding, and full trace-level observability and cost accounting, all built on framework-neutral domain contracts. This is a platform you own and push further, not just keep running. You'll contribute to architecture, raise the reliability and eval bar, and help define where a frontier agentic system goes. We hire engineers who can follow a problem wherever it leads, who know when deterministic logic or statistics beat an LLM and vice versa, and who care about the customer experience as much as the system underneath.

Requirements

• 3+ years building and shipping production software, with a track record of owning systems (not just features) and raising the quality bar for the people around you.
• Strong Python and CS fundamentals, including solid work with databases, queues, or real-time systems. The agent platform runs on Python (FastAPI, Pydantic, async), so you're fluent here or will be very quickly.
• Production agentic / LLM systems, not just calling an API: tool use, retrieval grounding, structured outputs, multi-turn state and continuity, streaming, evals, and designing for non-deterministic behavior. Having owned an agent runtime or orchestration layer end to end is a strong signal.
• Architectural judgment for AI systems: you keep domain logic decoupled from a fast-moving vendor framework, make build-vs-adopt calls deliberately, and know why that matters when the framework landscape shifts every quarter.
• Judgment about when to use an LLM and when not to: you reach for deterministic logic, retrieval, or statistics when they're more reliable, cheaper, or more reproducible, and you can tell which is which.
• AI-native workflow: you use AI coding agents (Claude Code, Cursor) as a real part of how you build, prompting for scaffolding, reviewing output critically, and knowing when to push back.
• Strong product sense and judgment. You care about the user experience and about system correctness in equal measure.
• Self-management and a continuous-improvement mindset. We don't over-prescribe how the work gets done.

Nice To Haves

• Experience with agent frameworks or orchestration: Microsoft Agent Framework, LangGraph, AutoGen, or a runtime you built yourself.
• RAG and tool-use platforms (retrieval over tools and APIs, OpenAPI-driven tool generation, MCP).
• LLM evals and observability: designing them, running them, and acting on the signal, with tracing and cost tooling like Langfuse or OpenTelemetry GenAI.
• Cost engineering on LLM workloads (caching, batching, model routing, prompt compaction).
• Embedding-based retrieval or clustering (vector DBs, hybrid search, HDBSCAN, UMAP, and similar).
• Multi-tenant SaaS architecture: data isolation, per-tenant state, noisy-neighbor concerns.
• Full-stack / core-services depth: production React/TypeScript, and/or our core stack (Elixir/Phoenix with OTP, ClickHouse, Kafka). You won't live here day to day, but it helps where the agent platform meets the rest of the product.
• Time-series anomaly detection or drift monitoring; recommendation or ranking systems with user-feedback loops.
• Spec-driven development, writing or working from specs that drive both human and AI implementation.
• Contributing to developer experience or agentic infrastructure.
• Technical leadership on an engineering team.
• Open source contributions.

Responsibilities

• Design, build, and operate the agent platform end to end, from the API surface through the runtime, tools, retrieval, persistence, and observability.
• Build LLM/agent features that ground reliably in customer data, with the streaming, retries, evals, and graceful degradation required to hold them to a production reliability bar.
• Pick the right tool for each signal (retrieval, deterministic logic, structured outputs, statistics, or an LLM), and combine them well.
• Treat evals, cost-per-call, and latency as first-class.
• Work in a spec-driven, agent-assisted flow, reading and contributing to PRDs that drive both human and AI implementation.
• Contribute to the team's agentic infrastructure (AGENTS.md, CLAUDE.md, DESIGN.md, slash commands, architectural rules) so AI tooling understands our codebase as well as the humans do.
• Review code for architectural consistency and reliability, including making sure agent-generated code respects the same boundaries and framework-neutral contracts that human-written code does.
• Raise the bar around you: set the patterns, write the specs and evals others build on, and level up the engineers (and agents) working in the platform.

Benefits

• We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other characteristic protected by applicable law. All qualified applicants will receive consideration for employment.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed