Software Engineer I - LLM Observability

Honeycomb.io

22h • $160,000 - $180,000 • Remote

About The Position

Honeycomb is hiring a Software Engineer I for its AI observability team. The team helps customers understand what AI agents are doing, what they cost, where errors occur, and how to debug individual conversations. It brings LLM and agent observability together, spanning the data foundation and product experiences built on top of it. You will work in a small, cross-functional team with a dedicated product manager and designer to ship production features.

The role delivers AI observability capabilities end to end, from data ingestion to in-product visualizations. It contributes to the data model and infrastructure for observing agents and their ecosystems, and builds systems to analyze and surface AI-specific signals such as cost, token usage, errors, and anomalies. The team works through experimentation and fast feedback loops, sharing both successes and failures.

Honeycomb is a fully distributed company that emphasizes autonomy, accountability, and delivery. The position includes joining the on-call rotation after about 90 days. The stated base salary range is $160,000–$180,000 USD, based on experience.

Requirements

• At least 3 years of software engineering experience.
• Experience shipping customer-facing features at a SaaS company, beyond internal tooling.
• Full-stack range, with backend proficiency in Go and/or Python and frontend proficiency in TypeScript and React.
• A track record of working with product managers and designers.
• Regular, hands-on use of AI tools in day-to-day engineering work.
• Experience being on call.
• A track record of navigating ambiguity in a fast-moving field where requirements shift and experimentation is a core part of the job.

Nice To Haves

• Experience building LLM- or agent-powered features in production.
• Hands-on experience with observability tooling, such as OpenTelemetry.

Responsibilities

• Deliver AI observability capabilities end to end, from ingestion to in-product visualizations.
• Contribute to the data model and infrastructure used to observe agents and their ecosystems across the Honeycomb product.
• Build systems that analyze and surface AI-specific signals, including cost, token usage, errors, and anomalies.
• Collaborate with product and design partners to build usable, intuitive features that support meaningful user workflows.
• Share learnings about exploring and understanding AI system behavior across conversations and workloads.
• Iterate through fast feedback loops and experimentation, and share both wins and failures.
• Join the on-call rotation after about 90 days.

Benefits

• Generous equity with an employee-friendly stock program.
• Transparent, experience-relative pay levels.
• Unlimited PTO.
• Home office, co-working, and internet stipend.
• Full benefits coverage for employees, with additional coverage available for dependents.
• Up to 16 weeks of paid parental leave, regardless of path to parenthood.
• Annual development allowance.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level
• Education Level: No Education Listed