Grafana and Obserability Engineer

DTCC • Coppell, TX

3d • Hybrid

About The Position

We are seeking a highly skilled Senior Observability Engineer to join our Observability Engineering team. This role is responsible for designing, implementing, administering, and automating enterprise observability solutions with a primary focus on Grafana, OpenTelemetry, monitoring, alerting, and telemetry management. The ideal candidate has strong experience building and operating observability platforms at scale, automating infrastructure through Terraform, and enabling application and infrastructure teams to adopt standardized observability practices. This role will play a key part in modernizing observability capabilities and driving migration from legacy monitoring tools to Grafana-based solutions.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
• 5+ years of experience in observability, monitoring, operations, or platform engineering.
• Hands-on experience administering Grafana in large-scale enterprise environments.
• Strong experience with Terraform and Infrastructure as Code practices.
• Experience implementing monitoring, alerting, logging, and distributed tracing solutions.
• Experience with OpenTelemetry concepts, instrumentation, and telemetry pipelines.
• Strong Linux and cloud platform administration skills.
• Experience with scripting and automation using Python, PowerShell, Bash, or similar languages.
• Knowledge of operational excellence, reliability engineering, and incident management practices.

Nice To Haves

• Experience migrating from tools such as Splunk, Dynatrace, AppDynamics, New Relic, OpenText OBM, or similar platforms.
• Experience with Grafana Alloy, Tempo, Loki, Mimir, or Prometheus.
• Experience operating observability platforms in AWS environments.
• Knowledge of Kubernetes, containers, and cloud-native observability.
• Experience designing enterprise observability strategies and governance models.
• Familiarity with CI/CD platforms and DevOps practices.

Responsibilities

• Administer and support Grafana Cloud and on-premises Grafana deployments.
• Design and implement enterprise observability solutions for metrics, logs, traces, synthetic monitoring, and alerting.
• Establish and maintain observability standards, best practices, and governance processes.
• Configure and manage Grafana data sources, alerting, RBAC, folders, teams, and integrations.
• Ensure platform scalability, reliability, resiliency, and operational excellence.
• Develop and maintain Terraform modules for Grafana infrastructure and configuration management.
• Automate onboarding of applications, infrastructure, dashboards, alerts, and data sources.
• Build self-service capabilities that reduce manual operational effort and improve adoption.
• Integrate observability capabilities into CI/CD and infrastructure provisioning workflows.
• Design meaningful monitoring and alerting strategies based on service health and business-critical workflows.
• Implement and optimize alerting standards to reduce noise and improve signal quality.
• Support incident response, troubleshooting, root cause analysis, and post-incident reviews.
• Drive continuous improvement of operational visibility and platform health.
• Implement and support OpenTelemetry instrumentation across applications and infrastructure.
• Establish standards for logs, metrics, traces, and telemetry collection.
• Support telemetry pipelines, agent deployments, and data collection strategies.
• Assist teams with instrumentation design and observability adoption.
• Support migration initiatives from legacy observability platforms to Grafana.
• Analyze existing monitoring, alerting, logging, and tracing implementations and recommend modernization approaches.
• Develop reusable migration patterns, automation, and engineering standards.
• Partner with application teams to accelerate adoption of enterprise observability capabilities.
• Work closely with application development, infrastructure, cloud, and SRE teams.
• Provide technical leadership and mentoring to engineers across the organization.
• Contribute to observability architecture, strategy, and roadmap development.
• Promote observability as a core engineering practice across the enterprise.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior