Director of Observability

Kestra Holdings•Tempe, AZ

About The Position

We are seeking a Director of Observability to stand up a brand-new observability and reliability practice from the ground up at Kestra Holdings. This is a working leadership role — the Director will be expected to be hands-on in the early phase: selecting and configuring tooling, writing instrumentation standards, building the first dashboards and alerting pipelines, and personally running incident command for major events while the team and platform mature. This is a newly created leadership role reporting directly to the Head of IT Infrastructure & Cybersecurity. The Director will start with two direct reports — a to be hired Senior Observability Architect (India-based) and a future US-based Observability/Reliability Engineer — and will be expected to scale the team over time as the practice and service catalog grow.

Requirements

10+ years in observability, SRE, platform engineering, or infrastructure operations roles, with 3+ years in a people leadership capacity (Director or Sr. Manager level).
Demonstrated experience building an observability or SRE practice from scratch — tool selection, instrumentation rollout, first SLOs, and standing up incident command.
Strong hands-on technical depths must be willing and able to write code/IaC, configure platforms, build dashboards, and run incidents personally, not just delegate.
Deep expertise across the observability stack: metrics (Prometheus, Datadog, Azure Monitor), log aggregation (Elastic/OpenSearch, Log Analytics, Splunk), distributed tracing (OpenTelemetry, Jaeger, Datadog APM), and profiling.
Proven experience defining SLIs/SLOs, error budgets, and toil reduction programs.
Hands-on experience with incident management platforms (PagerDuty, xMatters) and Jira / Jira Service Management integration for ticketing and workflow.
Experience leading distributed teams across US and India time zones.
Experience operating in a regulated industry (financial services, healthcare, or similar) with familiarity with compliance frameworks (NIST CSF, SOC 2, SEC, FINRA).
Excellent communication skills — able to present reliability posture, incident retrospectives, and risk to executive and board-level audiences.
Experience with IaC (Terraform, Bicep, ARM), CI/CD pipelines, and embedding observability-as-code into modern DevOps practices.

Responsibilities

Define and execute the observability strategy for Kestra Holdings, aligned with business objectives, regulatory requirements, and the enterprise technology roadmap.
Personally lead the initial build-out of the observability platform across metrics, logs, traces, profiles, and alerting — including tool evaluation, POCs, architecture, deployment, and configuration (e.g., Azure Monitor/Log Analytics, Datadog, Grafana, OpenTelemetry, Elastic/Splunk).
Work with and enforce existing instrumentation standards (OpenTelemetry, structured logging, distributed tracing) across infrastructure and application teams.
Build the first generation of dashboards, SLO scorecards, and a single pane of glass for Tier-1 service health — rolling up sleeves alongside the Sr. Architect and engineer.
Operate the firm's end-to-end incident management lifecycle — detection, response, escalation, communication, and blameless post-incident review.
Stand up on-call schedules, escalation policies, and runbook-driven triage for Sev1–Sev4 incidents via PagerDuty/xMatters or equivalent.
Serve as primary incident commander for major incidents during the initial build phase, transitioning command responsibilities to senior team members as the practice matures.
Integrate the incident lifecycle with Jira / Jira Service Management (JSM) for ticketing, change correlation, and remediation tracking; partner with Cybersecurity so incidents run once, not separately by Infra and Cyber.
Facilitate post-incident reviews (PIRs), track remediation items in Jira, and report trends to leadership.
Drive adoption of SRE principles across the firm: SLI/SLO definition, error budget policy and enforcement, toil identification and automation, and operational readiness reviews.
Establish release of reliability gates and embed reliability into the service lifecycle from design through production.
Partner with Cloud & Platform Engineering, Cybersecurity, and application teams to ensure all services are fully instrumented, measurable, and integrated into the firm's SLO and incident frameworks.
Ensure observability and incident management practices align with NIST CSF 2.0 maturity targets and support the firm's cybersecurity roadmap.
Partner with the Cybersecurity team to integrate observability data with Jira/JSM, CMDB, and SIEM for enriched context during incidents.
Support regulatory and audit requirements appropriate for a SEC-regulated financial services firm (e.g., logging retention, evidentiary integrity, access controls on telemetry data).
Directly lead and mentor a small, high-leverage team of two to start: a Senior Observability Architect (India-based) and a US-based Observability/Reliability Engineer.
Operate as a player-coach — splitting time between strategic leadership, hands-on engineering, and direct mentorship of the senior architect.
Build a multi-year workforce plan and talent pipeline to scale the team as the platform, service catalog, and 24/7 coverage needs grow.
Foster a culture of blameless learning, operational excellence, and engineering-led reliability across US and India hours of coverage.
Serve as the primary technical liaison for observability and incident management vendors (e.g., Datadog, PagerDuty/xMatters, Grafana Labs, Elastic/Splunk, Atlassian).
Represent Observability & Reliability in the Architecture Review Board, IT Change Management Board, and incident command forums.
Provide regular reporting to SVP and executive leadership on reliability KPIs (MTTD, MTTA, MTTR, SLO compliance, alert signal-to-noise), incident trends, and strategic initiatives.