Software Engineer, Agent Platform

Retool•San Francisco, CA

3d•Hybrid

About The Position

Nearly every company in the world runs on custom software for critical operations like tracking performance metrics, handling support workflows, building admin dashboards, and countless processes you might never have thought of. But most companies don't have the resources to properly invest in these tools, leading to a lot of old, clunky internal software, or worse, teams still stuck in manual and spreadsheet workflows. AI has changed who gets to build software. The definition of "developer" now includes analysts, operators, and domain experts creating solutions directly—and the tools they reach for are multiplying by the week. That's both an opportunity and a challenge: as more people build with more AI tools, the risk of shipping ungoverned software into production grows just as fast. At Retool, we're building the platform that makes all of it safe to ship. Build with any AI tool you want, then deploy into one place that connects to your real business data, enforces enterprise policies automatically, and lets teams create once and reuse everywhere with shared, trusted components. The cost of building software has collapsed. The cost of governing it hasn't—and that's the problem we solve. Developers and domain experts have already automated over 100 million hours of work on our platform, freeing them to focus on creative problem-solving and strategic work that drives real business value. The people closest to the problem can now build the software to solve it, safely, and within enterprise guardrails. Let's build the future together.

Requirements

6+ years of professional engineering experience, with ownership over complex systems in production.
Demonstrated experience owning AI/LLM behavior beyond basic integration, including mitigation of variance and failure modes.
Comfort reasoning about probabilistic systems and tradeoffs (quality vs. cost, recall vs. precision, speed vs. robustness).
Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
Strong product intuition—you care deeply about what “good” looks like even when outputs are non-deterministic.
Ability to operate independently in ambiguous problem spaces and set quality standards others rely on.
Strong opinions, weakly held—you iterate quickly and adjust based on evidence and observed runtime behavior.
Work across the stack (e.g., TypeScript, Node.js, React).

Nice To Haves

Experience with RAG, agentic systems, or tool-using models in production.
Familiarity with vector databases, embeddings, or retrieval pipelines.
Exposure to fine-tuning, model routing, or post-training techniques.
Experience building shared AI infrastructure used by multiple teams.
History of mentoring engineers on designing for non-determinism and evaluation-driven development.

Responsibilities

Own model-driven behavior in production systems, working across product, infrastructure, and evaluation layers.
Own the behavior of AI-powered features across multiple product surfaces, including quality, safety, variance, and failure modes.
Design and evolve prompting, retrieval, routing, and tool-use strategies that embrace non-determinism while bounding its downside.
Build and maintain evaluation systems that measure model performance using statistical signals, distributions, and trends—not just pass/fail tests.
Detect, diagnose, and resolve non-deterministic failures such as hallucinations, partial correctness, instruction drift, or sensitivity to context changes.
Define and implement guardrails, fallbacks, and degradation paths that keep systems useful even when models behave unexpectedly.
Partner with product and infra teams to decide when probabilistic behavior is “good enough” to ship—and when it isn’t.
Influence model selection, model behavior, and tool design to balance quality, cost, latency, and robustness for real user workflows.
Set standards that others build on.
Act as the final owner of AI behavior and quality before features reach users.