Senior Software Engineer II - LLM Observability

Honeycomb.io

1d • $183,340 - $206,000 • Remote

About The Position

Honeycomb is hiring a Senior Software Engineer II for its AI observability team. The team helps customers understand what AI agents are doing, what they cost, and where errors occur, and provides conversation-level insight for debugging. It brings LLM and agent observability together, owning both the data foundation and product experiences built on it. The role is part of a small, cross-functional group that includes a dedicated product manager and designer, and ships features used by customers in production.

The engineer will own end-to-end development of AI observability capabilities, from ingestion through in-product visualizations; architect and evolve the data model and infrastructure for observing agents and their ecosystems; and build systems to analyze and surface AI-specific signals such as cost, token usage, errors, and anomalies. The role also involves collaborating with product and design, teaching customers best practices for understanding AI system behavior, and working iteratively through experimentation and feedback. The position is fully distributed. On-call rotation participation begins after approximately 90 days.

Requirements

• Significant software engineering experience, with a record of shipping and owning features end to end.
• Experience shipping customer-facing features at a SaaS company, beyond internal tooling.
• Full-stack range, with proficiency in backend development using Go and/or Python and frontend development using TypeScript and React.
• A track record of working closely with product managers and designers.
• Regular, hands-on use of AI tools in day-to-day engineering work.
• Experience being on call.
• A track record of navigating ambiguity in a fast-moving field where requirements shift and experimentation is a core part of the job.

Nice To Haves

• Experience building LLM- or agent-powered features in production.
• Hands-on experience with observability tooling, such as OpenTelemetry.

Responsibilities

• Own end-to-end development of AI observability capabilities, from ingestion to in-product visualizations.
• Architect and evolve the data model and infrastructure used to observe agents and their ecosystems across the Honeycomb product.
• Build systems that analyze and surface AI-specific signals, including cost, token usage, errors, and anomalies.
• Collaborate with product and design partners to build usable, intuitive features that enable meaningful user workflows.
• Teach best practices for exploring and understanding AI system behavior across conversations and workloads.
• Iterate through fast feedback loops and experimentation, and share both wins and failures.
• Participate in the on-call rotation after approximately 90 days.

Benefits

• Generous equity with an employee-friendly stock program.
• Transparent pay based on levels relative to experience.
• Unlimited PTO.
• Home office, co-working, and internet stipend.
• Full benefits coverage for employees, with additional coverage available for dependents.
• Up to 16 weeks of paid parental leave, regardless of path to parenthood.
• Annual development allowance.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed