Observability Engineer

Goosehead Insurance•Westlake, OH

About The Position

The Observability Engineer builds and improves capabilities for understanding the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry. The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool. The core outcome is to make production health visible, alerts actionable, and failures easier to diagnose.

Requirements

At least 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4–5 years is preferred.
Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or a related field preferred; equivalent technical experience is considered.

Nice To Haves

Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
Experience establishing SLIs/SLOs or service-health measures for production applications.
Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
Familiarity with ITIL Incident Management and Problem Management practices.

Responsibilities

Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry to instrument metrics, logs, traces, and custom telemetry.
Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that clarify availability, latency, errors, dependencies, and customer impact.
Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts, and ensure alerts support effective NOC triage and escalation.
Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
Maintain service-based error budgets used by development teams to guide feature work.