AI Research Engineer

Traversal•New York, NY

3d•Onsite

About The Position

As an AI Research Engineer at Traversal, you'll build the systems our research team runs on. Our agents autonomously diagnose and resolve production incidents for some of the world's largest enterprises, and improving them is an experimental problem: form a hypothesis about agent behavior, test it against real incident data, and ship what works. The bottleneck is rarely ideas. It's the infrastructure to run the experiment, trust the result, and scale the prototype. That infrastructure is what you own. You'll build the harnesses that let researchers evaluate agents against real customer incidents, the tooling that makes agent behavior legible enough to debug, and the systems that take a promising prototype from one run to thousands. You'll design experiments alongside researchers rather than only implementing them, and you'll carry the results across the line into production. This role sits on the research team, and it is an engineering role. We are looking for a strong backend engineer with real experimentation instincts — someone who has always run experiments on the side and wants that to be the job. No PhD required. If you want to work at the frontier of LLM agents while still building serious systems, this is the seat.

Requirements

5+ years of software engineering experience, with a strong focus on backend development and experience building scaled systems.
Hands-on experimentation experience — running experiments, evaluating results, and iterating on them. Side projects and personal work count.
Proven ability to lead in fast-paced startup environments, with limited resources, shifting priorities, and minimal structure.
Strong experience with Python and web frameworks such as FastAPI.
Experience deploying applications on AWS, working with ECS or Kubernetes, Postgres for data storage, and S3 for large-scale object storage.
Familiarity with observability and monitoring tools to ensure the system is running smoothly and efficiently.

Nice To Haves

A research engineer background, or prior experience as the engineer embedded in a research team.
Experience building or operating agentic AI systems.
Background in large-scale, complex, data-driven applications.
Familiarity with AI or LLM-powered products, and with evaluation or benchmarking tooling.

Responsibilities

Build and scale the infrastructure behind the team's research — experiment harnesses, evaluation pipelines, and trajectory tooling — so that ideas can be tested quickly and results can be trusted.
Take promising research prototypes and make them fast, reliable, and production-ready, closing the gap between a one-off experiment and a capability that ships to enterprise customers.
Partner with researchers to design experiments — framing the hypothesis, choosing the measurement, and making sure the result is real before it drives a decision.
Work closely with AI engineers, infrastructure teams, and product leads to bring research into production and close the loop between experimentation and impact.
Track developments in LLMs, agent architectures, and evaluation methodology, translating insights into actionable improvements for Traversal's domain.
Prototype and evaluate prompting strategies, reasoning workflows, and tool-use policies for agents operating on large-scale observability data and complex troubleshooting workflows. Ship improvements to production.