Senior Splunk Observability Engineer

Cynet Systems•Dallas, TX

About The Position

The Observability Engineer designs, implements, and continuously improves enterprise observability, service intelligence, and event intelligence capabilities across applications, infrastructure, cloud platforms, and digital services. The role serves as a subject matter expert for Client Enterprise and Client IT Service Intelligence (ITSI), leading the development of service intelligence, KPI frameworks, service dependency models, event correlation strategies, and operational analytics that turn telemetry into actionable business and technology insights. The position partners with application, infrastructure, network, cloud, and operations teams to translate business requirements into observability and service intelligence solutions. It supports platform administration and architecture initiatives and troubleshoots complex performance, availability, reliability, data quality, and service intelligence issues using telemetry, metrics, logs, traces, and operational data. The role also develops and maintains observability documentation, standards, onboarding procedures, and technical knowledge resources.

Requirements

  • At least 5 years of hands-on experience supporting observability, application performance monitoring, service intelligence, event management, or related operational visibility functions.
  • Advanced experience with Client Enterprise, Client ITSI, service intelligence, event management, AIOps, or comparable observability platforms.
  • Advanced knowledge of Client SPL, dashboards, reports, data models, correlation searches, knowledge objects, lookups, and operational analytics.
  • Experience designing and administering service models, KPIs, service analyzers, glass tables, notable event aggregation policies, and adaptive thresholding capabilities.
  • Experience developing custom data onboarding, parsing, normalization, enrichment, and telemetry integration solutions.
  • Strong understanding of observability fundamentals, including metrics, logs, traces, OpenTelemetry, SLIs, SLOs, distributed tracing, and actionable alerting.
  • Knowledge of Client architecture, including search heads, indexers, forwarders, clustering concepts, and data lifecycle management.
  • Strong analytical, troubleshooting, problem-solving, and stakeholder communication skills.
  • At least 5 years of experience in observability, application performance monitoring, service intelligence, platform engineering, or related disciplines is required.

Nice To Haves

  • Bachelor's degree in Information Technology, Computer Science, Engineering, Information Systems, or a related field; equivalent experience may be considered.
  • Experience implementing, administering, supporting, or architecting Client Enterprise and Client ITSI environments.
  • Experience developing service models, KPI frameworks, event correlation logic, operational analytics, custom Client content, or data onboarding solutions.
  • Experience supporting cloud-native, hybrid cloud, distributed, or large-scale enterprise environments.
  • Client Core Certified Power User, Client Enterprise Certified Admin, Client ITSI certifications, or comparable certifications.

Responsibilities

  • Develop and maintain Client ITSI service models, service hierarchies, dependency maps, KPI frameworks, health scores, and service health methodologies for business-critical services.
  • Develop advanced Client SPL searches, KPI calculations, service intelligence content, analytics, and operational reporting to monitor service health and operational performance.
  • Design, develop, and maintain Client Technology Add-ons (TAs), data onboarding solutions, field extractions, parsing logic, transforms, normalization processes, and telemetry integrations.
  • Configure and optimize Client ITSI event aggregation, notable event correlation, suppression, enrichment, adaptive thresholding, and event intelligence capabilities.
  • Design and maintain Client ITSI glass tables, executive and operational dashboards, service analyzers, and other service-health visualizations.
  • Define and drive observability practices and standards, including instrumentation strategies, telemetry requirements, monitoring coverage, and naming and tagging conventions.
  • Design, implement, and maintain observability, service intelligence, and monitoring solutions across applications, infrastructure, cloud services, databases, and supporting technologies.
  • Design, develop, and maintain data onboarding solutions that ingest, normalize, enrich, and validate telemetry from enterprise technology platforms.
  • Define and support SLIs, SLOs, KPIs, and service health metrics aligned with business outcomes.
  • Support Client platform administration, including search optimization, app lifecycle management, knowledge object management, platform performance tuning, and observability architecture initiatives.
  • Partner with application, infrastructure, network, cloud, and operations teams to translate business requirements into service intelligence and observability solutions.
  • Troubleshoot performance, availability, reliability, data quality, and service intelligence issues using telemetry, metrics, logs, traces, and operational data.
  • Develop and maintain observability documentation, standards, onboarding procedures, and technical knowledge resources.
  • Perform all essential aspects and functions of the job and any other specific job requirements.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service