Site Reliability Engineer

TQL • Cincinnati, OH

12d

About The Position

The Site Reliability Engineer is the technical leader responsible for designing, implementing, and operating enterprise observability platforms and standards. The role drives strategy for logging, metrics, distributed tracing, monitoring, alerting, and self-healing automation across application, infrastructure, and AI-enabled environments. It establishes visibility into AI and agentic workflows, including model interactions, retrieval and tool execution, performance, quality, cost, and operational risk.

The engineer partners with development teams, platform engineers, AI platform teams, and enterprise architects to build observability and traceability into deployment pipelines and solution architectures. The role leads improvements in monitoring maturity and operational resilience to enable faster incident detection, diagnosis, and resolution. It also owns and evolves the observability platform strategy and roadmap, including Datadog and SolarWinds, and integrates observability with ServiceNow, CMDB, and Service Mapping.

Requirements

• Bachelor’s degree in computer science, engineering, or an equivalent combination of education and experience.
• 1–3 years of site reliability engineering, DevOps, or platform engineering experience.
• Hands-on experience building and operating enterprise observability platforms.
• Ability to demonstrate success implementing monitoring, logging, and tracing at scale across complex environments.
• Deep expertise with Datadog and SolarWinds monitoring platforms.
• Strong experience designing observability for cloud, hybrid, and on-premises environments, including Azure, Kubernetes, and VMs.
• Expertise in logging platforms, APM, distributed tracing, and infrastructure monitoring.
• Working knowledge of generative AI and LLM application architectures, including copilots, agents, orchestration frameworks, RAG, vector databases, model gateways, and tool-based integrations.
• Experience with OpenTelemetry or comparable instrumentation standards and applying trace context across AI, application, API, and infrastructure dependencies.
• Understanding of AI observability concepts, including prompt and model versioning, token and cost analysis, latency decomposition, quality evaluation, drift detection, guardrails, and human feedback signals.
• Ability to balance diagnostic trace depth with privacy, security, retention, and cost requirements when collecting AI telemetry.
• Strong proficiency in automation and scripting, including Python and PowerShell.
• Experience implementing observability in CI/CD pipelines, such as GitHub Actions, Azure DevOps, or GitLab.
• Strong understanding of infrastructure-as-code and configuration management, such as Terraform and Ansible.
• Experience with container and orchestration platforms, including Docker and Kubernetes.
• Strong understanding of networking, operating systems, and application architectures.
• Experience with event-driven monitoring, anomaly detection, and predictive alerting.
• Excellent troubleshooting, root cause analysis, and incident management skills.
• Strong documentation, architecture diagramming, and platform governance capabilities.
• Ability to lead cross-functional reliability initiatives and influence engineering standards.
• Applicants must not require employment visa sponsorship now or in the future; sponsorship is unavailable.

Nice To Haves

• Experience implementing observability and distributed tracing for AI-enabled applications, LLM workflows, or agentic systems.

Responsibilities

• Lead the design and implementation of enterprise observability standards for logging, metrics, tracing, and alerting.
• Own and evolve the observability platform strategy and roadmap, including Datadog and SolarWinds.
• Define and implement instrumentation standards for applications, APIs, and infrastructure to ensure consistent telemetry collection.
• Partner with development teams to embed observability into CI/CD pipelines as a deployment requirement.
• Design automated monitoring, alerting, and self-healing patterns to improve service reliability and availability.
• Develop reusable observability frameworks, libraries, and deployment templates for engineering teams.
• Establish monitoring-as-code, alerting-as-code, and dashboard-as-code standards.
• Design SLIs, SLOs, and error budgets with product and engineering teams.
• Ensure monitoring coverage across cloud, on-premises, containerized, and hybrid environments.
• Implement distributed tracing and log correlation for end-to-end transaction visibility.
• Define observability and tracing standards for AI-enabled applications, copilots, and autonomous or agentic workflows, including model calls, prompts and responses, retrieval steps, tool invocations, latency, token consumption, cost, and failure states.
• Implement end-to-end AI tracing that correlates requests across orchestration layers, models, RAG, vector stores, APIs, tools, and downstream services.
• Establish monitoring and evaluation patterns for AI response quality, hallucination risk, model or prompt drift, guardrail violations, safety events, and degraded user outcomes.
• Define telemetry governance for AI workloads, including sensitive-data redaction, access controls, retention, sampling, and secure handling of prompts, responses, and trace data.
• Serve as the escalation point for complex monitoring, performance, and reliability issues.
• Collaborate with enterprise and platform architects to incorporate observability requirements into reference architectures.
• Drive root cause analysis and reliability improvements following major incidents.
• Develop platform governance standards for monitoring, alert thresholds, and on-call practices.
• Integrate observability with CMDB and Service Mapping for accurate service context and impact analysis.
• Partner with DevOps and platform engineering teams to improve deployment reliability and pipeline automation.
• Continuously tune alerts to reduce noise and improve signal quality.
• Support capacity planning, performance engineering, and load testing.
• Engage with Datadog, SolarWinds, and ServiceNow vendors and user communities to stay current with platform capabilities.
• Develop training and enablement materials on observability best practices.
• Contribute to reliability engineering practices, including chaos testing, fault injection, and resiliency validation.
• Improve incident response, post-incident reviews, and operational readiness.
• Integrate observability platforms with ServiceNow for incident, event, change, and problem automation.
• Define observability policy aligned with CMDB governance, service modeling, and configuration management standards.

Benefits

• Career advancement opportunities and the ability to write your own paycheck.
• Sales support teams assist with accounting, after-hours calls, and specific needs to help manage work-life balance.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level