Principal Software Engineer

HubSpot

7d • $313,800 - $502,080 • Hybrid

About The Position

The Observability team owns the internal platform that gives every HubSpot engineer real visibility into how their systems behave in production. We build and operate the distributed tracing, metrics, alerting, and logging infrastructure that spans hundreds of microservices, billions of daily events, and thousands of engineers who depend on that signal to ship reliably. We are now investing in the next generation of this platform. As HubSpot deploys AI agents and ML-powered features across the product, the team is building the tracing and telemetry primitives that make it possible to understand, debug, and trust what those systems are doing in production. This is greenfield, technically interesting work at a scale few companies operate at and we are looking for a Principal Engineer to help lead it. We are seeking a Principal Software Engineer to be the technical anchor for HubSpot’s Observability platform. This role sits at the intersection of large-scale distributed systems, developer platform design, and AI observability. A big part of this role is working horizontally across a large engineering org and setting patterns and standards that make it easier for teams to instrument, alert on, and reason about their services. You will also shape how we trace and understand our growing fleet of AI agents and ML systems in production: a technically distinct and increasingly critical problem.

Requirements

• Proven experience building observability or telemetry tooling for internal engineering teams, rather than simply consuming it. You understand how to architect developer platforms that serve thousands of engineers across a large organization, backed by deep operational instincts and hard-earned expertise.
• Deep expertise navigating trade-offs in telemetry pipeline design across high-cardinality data, dynamic sampling, query latency, retention economics, and data fidelity. Strong technical fluency with OpenTelemetry, distributed tracing engines, metric backends, and large-scale log ingestion infrastructure.
• Proven track record linking telemetry signals directly to automated operational workflows. You have designed architecture for real-time telemetry triggers that power automated remediation, dynamic runbooks, or AI-assisted root-cause diagnosis across microservices environments.

Responsibilities

• Define the patterns and evolution of HubSpot’s core telemetry platform — distributed tracing, metrics, and structured logging — at a scale that spans hundreds of services and billions of daily events. Set the standards for how instrumentation is done across a large, polyglot engineering organization.
• Lead the technical strategy for tracing and understanding AI agents and ML-powered systems in production. Define the primitives, telemetry standards, and debugging workflows that help product engineers understand what their models and agents are doing — and build trust in those systems over time. This is greenfield and consequential work.
• Architect telemetry pipelines and storage systems that handle high-cardinality data at high throughput without blowing up cost or query latency. Make principled tradeoffs between sampling, fidelity, retention, and developer ergonomics.
• Ship production code. Lead design reviews and take high-impact initiatives end-to-end, from prototype to production system at scale. Stay close to the systems you build and be the person who can debug the hardest problems when they surface.
• Design the instrumentation APIs and libraries that product engineers reach for, making correct observability the path of least resistance. Drive OpenTelemetry adoption across a large, polyglot codebase. Build the tooling that turns raw telemetry into actionable signal for teams operating at speed.
• Define patterns for SLO/SLI design, alerting philosophy, and how teams graduate from reactive to proactive incident response. Push for simplicity in a domain that wants to get complicated, and consistency where tooling can drift across a large organization.
• Partner with infrastructure, platform, and product engineering teams to understand their signal gaps and close them. Influence technical strategy alongside engineering leadership, translating observability constraints and opportunities into product and operational decisions. Mentor senior engineers and tech leads, driving thoughtful design decisions and capturing learnings from major incidents and large-scale migrations.

Benefits

• The cash compensation below includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles.
• In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs).
• Some roles may also be eligible for overtime pay.
• Individual compensation packages are tailored to your skills, experience, qualifications, and other job-related reasons.
• Explore the benefits and perks HubSpot offers to help employees grow better.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed