Applied AI Systems Engineer

Neon Health•San Francisco, CA

About The Position

Neon is hiring an Applied AI Systems Engineer to improve the real-world behavior and reliability of its AI agent platform, which automates patient access workflows for pharmaceutical companies. This is an applied systems role focused on agent behavior, infrastructure, and debugging production systems—not model training or model research and development. The engineer will work across prompts, runtime debugging, agent simulation, evaluation and QA tooling, client data integration, and system monitoring. The role involves investigating real-world edge cases, validating agent behavior by inspecting outputs, and building tools and infrastructure that make AI systems more observable, testable, and scalable. Neon describes its environment as fast-paced, high-agency, and comfortable with ambiguity, with frequent team offsites and an intense, collaborative working style.

Requirements

  • 2–7 years of engineering experience working closely with LLMs or AI agents in production systems.
  • Proficiency in Node.js, TypeScript, and Python.
  • Experience with prompt engineering, LLM evaluations, and agent orchestration.
  • Strong debugging skills, including tracing runtime failures through logs and diagnosing issues in asynchronous or multi-step agent systems.
  • Ability to build data pipelines to normalize and convert unstructured data for AI systems.
  • Strong understanding of system design, including distributed systems and reliability and performance tradeoffs.
  • Experience using modern AI coding tools such as Cursor, GitHub Copilot, or Claude.
  • Strong documentation and testing discipline.
  • Proficiency with Git.
  • Ability to thrive with minimal structure and ambiguity, take initiative, and work independently to create value.
  • Willingness to investigate detailed real-world edge cases and improve system behavior through prompts, code, and evaluations.
  • Ability to learn quickly and adapt prior systems experience to LLM-specific issues.
  • Experience with LLM-based systems, such as writing prompts, chaining model calls, designing evaluations, or building agents that take actions or simulate behavior.
  • Experience creating internal tools or frameworks for QA, evaluations, or agent simulation, and a concern for making complex systems observable and testable.
  • Experience contributing to fast-paced product cycles involving AI behavior, latency, and user experience, including validating behavior by inspecting outputs.

Nice To Haves

  • Experience with multi-agent systems, TTS/NLP pipelines, or structured output validation.
  • Familiarity with testing frameworks, LangChain-style agent orchestration, or in-house evaluation harnesses.

Responsibilities

  • Trace and fix runtime bugs in AI agent systems and write regression tests.
  • Design evaluation datasets that simulate realistic workflows or red-team the system.
  • Build internal QA tools and agent simulation tooling.
  • Normalize and transform client data for integration with AI systems.
  • Set up automated testing and latency-tracking infrastructure.
  • Create dashboards and observability tools for agent behavior.
  • Expand the existing evaluation, testing, and agent simulation infrastructure.
  • Write and refine prompts, design evaluations, and investigate LLM outputs.
  • Improve agent performance by identifying real-world edge cases and creating tests to cover them.

Benefits

  • Frequent team offsites, spending one week each month at locations such as Tahoe, Squamish, Mendocino, the Santa Cruz Mountains, and Monterey.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service