Senior Observability Engineer

CoBank • Greenwood Village, CO

4d • Hybrid

About The Position

Are you a seasoned professional with a passion for observability, telemetry, and system reliability? Do you excel in a dynamic, collaborative, and inclusive work environment? At CoBank, we are seeking a Senior Observability Engineer to help design, implement, and scale enterprise observability capabilities across both on-premises and cloud environments. In this role, you will play a key part in defining and advancing CoBank's observability strategy, with an initial focus on instrumentation and telemetry for on-prem systems, evolving into cloud-native observability within AWS. You will partner closely with infrastructure and application teams to enable visibility across systems and services, ensuring reliable, performant, and measurable platforms. As a Senior Observability Engineer at CoBank, you will be responsible for implementing and optimizing observability solutions using platforms such as Splunk and Splunk Observability Cloud, while supporting modern telemetry standards including OpenTelemetry. You will also contribute to building scalable ingestion pipelines, improving signal quality, and enabling actionable insights across infrastructure and applications.

Requirements

• 5 years of experience in DevOps, SRE, cloud engineering, or IT operations required
• 3 years of experience implementing or supporting observability/monitoring solutions required
• Hands-on experience with Splunk (search, dashboards, alerting, ingest).
• Experience with OpenTelemetry or similar instrumentation frameworks.
• Experience supporting observability across hybrid environments (on-prem and cloud; AWS preferred).
• Experience integrating telemetry from enterprise systems such as infrastructure platforms, virtualization, or cloud services.

Nice To Haves

• Familiarity with telemetry pipelines or aggregators (e.g., OpenTelemetry Collector, Logstash, Fluentd) preferred.
• Experience supporting application teams with monitoring, APM, and instrumentation preferred.
• Exposure to open-source observability tools such as Prometheus, Grafana, or ELK stack preferred.
• Experience with Splunk Observability Cloud, Datadog, Dynatrace, or similar platforms preferred.
• Experience optimizing telemetry ingest, alerting strategies, or data retention is a plus preferred.
• Strong problem-solving skills and ability to work in a fast-paced environment preferred.
• Ability to apply independent judgment on most decisions and work with minimal guidance preferred.
• Demonstrates strategic thinking and ownership of technical outcomes preferred.
• Strong collaboration skills across engineering, operations, and development teams preferred.
• Enhances relationships and builds partnerships with cross-functional teams including infrastructure, application development, and security. Effectively communicates complex observability concepts to technical and non-technical audiences and helps guide teams in adopting best practices.
• Works on problems of diverse scope where analysis requires evaluation of multiple system signals and telemetry sources. Applies judgment to design scalable observability solutions and adapt existing approaches to improve visibility and reliability. Operate independently, with work reviewed at key milestones.
• Demonstrates strong knowledge of observability principles including metrics, logs, traces, and alerting strategies. Apply modern practices to improve system insight, performance, and reliability. Resolves complex issues through analysis of telemetry data and contributes to evolving observability standards across the organization.

Responsibilities

• Designs, implements, and maintains enterprise observability solutions across on-prem and AWS environments.
• Develops and enhances monitoring, alerting, logging, and tracing capabilities using Splunk and related platforms.
• Implements instrumentation standards using OpenTelemetry and other telemetry frameworks.
• Integrates telemetry from diverse systems (infrastructure, applications, and platforms) into centralized observability solutions.
• Collaborates with application teams to enable APM, distributed tracing, and performance monitoring.
• Builds and manages telemetry pipelines and collectors (e.g., OpenTelemetry Collector or similar tools).
• Monitors system health and proactively identify reliability and performance issues.
• Optimizes telemetry ingest, data quality, and alert effectiveness to reduce noise and cost.
• Troubleshoots issues across infrastructure, applications, and observability platforms.
• Partners with cross-functional teams to establish and promote observability best practices.
• Stays current with emerging trends and technologies in observability, APM, and SRE practices.

Benefits

• 15 days of vacation
• 10 paid sick days
• 11 paid holidays
• Competitive Compensation & Incentive
• Medical, Dental and Vision coverage
• Disability, AD&D, and Life Insurance
• Robust associate training and development with CoBank University
• Tuition reimbursement for higher education
• Outstanding 401k: up to 6% matching and additional 3% non-elective contribution & Student Loan Match
• Community Impact: United Way Angel Day, Volunteer Day and Associate Directed Contribution
• Associate Resource Groups: creating a culture of respect and inclusion
• Recognize a fellow associate through our GEM awards

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Associate degree