Senior Site Reliability Engineer

Hard Rock Digital•United States, FL

2d•Hybrid

About The Position

We are looking for a Senior Site Reliability Engineer who combines deep infrastructure expertise with a forward-thinking approach to AI-driven operations. In this role you will maintain and improve the reliability, scalability, and performance of our Java-based applications while pioneering the use of large language models (LLMs), agentic workflows, and intelligent automation to transform how we monitor, respond to, and prevent incidents. You will design and build autonomous and semi-autonomous AI agents that consume observability data, triage alerts, generate runbooks, automate incident response steps, and surface actionable insights—reducing toil and accelerating mean time to resolution. This is a hands-on engineering role for someone who is equally comfortable tuning a JVM, writing PromQL, and prototyping an agentic pipeline with tool-calling LLMs.

Requirements

Degree in Computer Science or a related field, or equivalent professional experience.
5+ years in SRE, DevOps, or similar infrastructure roles with experience managing large-scale, high-availability production systems.
3+ years hands-on experience managing production Kubernetes clusters, including deep understanding of architecture, networking, storage, and security.
Experience with cluster autoscaling (Karpenter), upgrades, and multi-cluster management.
Proficiency with kubectl, Helm, Kubernetes operators, and container orchestration troubleshooting.
Advanced expertise with the Grafana observability stack: dashboards, alerting, visualization, and Grafana Alloy for telemetry collection.
Proficiency in PromQL and experience with Loki for log aggregation and analysis.
Hands-on experience managing Java-based applications in distributed environments, including JVM tuning and optimization.
Cloud platform expertise (AWS preferred; GCP or Azure also valued).
Familiarity with Infrastructure as Code tools such as Terraform/Terragrunt or Ansible.
ArgoCD proficiency for GitOps workflows and continuous deployment.
Strong scripting abilities in Python, Bash, or Go, with experience building CI/CD pipelines and deployment automation.
Proven track record with on-call rotations, incident response, and root cause analysis.
1+ years of practical experience building or operating AI/LLM-powered tools, agents, or workflows in a production or production-adjacent context.
Demonstrated ability to design agentic systems that use tool calling, retrieval-augmented generation (RAG), or multi-step reasoning to accomplish operational tasks.
Experience integrating LLM APIs (e.g., Anthropic Claude, OpenAI, or open-source models) into backend services or automation pipelines.
Familiarity with at least one agentic orchestration framework or workflow engine (LangChain, LangGraph, CrewAI, n8n, Temporal, or equivalent).
Understanding of prompt engineering best practices, including structured outputs, system prompts, and few-shot examples.
Familiarity with AI-assisted coding tools (Claude Code, Codex, Cursor) and their integration into engineering workflows.
Experience building or consuming MCP (Model Context Protocol) servers to expose internal tools to AI agents.
Awareness of AI safety, hallucination mitigation, and human-in-the-loop design patterns for autonomous systems.
Strong communication skills (written and verbal) with the ability to translate complex AI and infrastructure concepts for diverse audiences.
Proactive problem-solver with a bias toward automation and continuous improvement.
Ability to mentor junior team members on both traditional SRE practices and emerging AI-driven approaches.
Positive attitude and openness to constructive feedback.

Nice To Haves

Hands-on experience with vector databases (Pinecone, Weaviate, pgvector) for RAG-based knowledge retrieval.
Experience with LLM evaluation frameworks (e.g., Galileo, LangSmith, Braintrust) for monitoring agent quality in production.
Contributions to open-source AI/ML or SRE tooling projects.
Background in data engineering or ML pipelines that complements SRE responsibilities.

Responsibilities

Ensure the availability, reliability, and performance of high-traffic Java-based applications in a distributed environment.
Troubleshoot and resolve complex issues across production and non-production environments.
Participate in pre- and post-deployment performance testing and monitoring to continuously improve application performance.
Optimize Java application performance with a focus on JVM tuning, efficient resource utilization, and horizontal scaling.
Deploy and manage the Grafana stack (Grafana, Prometheus, Loki, Mimir, Alloy) to deliver real-time monitoring, logging, and alerting.
Implement and refine observability strategies that enhance visibility into application and infrastructure health.
Create and maintain dashboards, alerts, and log queries for comprehensive system health monitoring.
Integrate AI/ML models into the observability pipeline for anomaly detection, predictive alerting, and intelligent alert correlation and noise reduction.
Design, build, and operate agentic AI workflows that automate operational tasks such as alert triage, root cause analysis, runbook execution, and incident summarization.
Develop tool-calling LLM agents that interact with infrastructure APIs (Kubernetes, Grafana, Jira, Slack, PagerDuty) to execute diagnostic and remediation actions autonomously or with human-in-the-loop approval.
Build and maintain MCP (Model Context Protocol) servers and integrations that expose internal systems as tool surfaces for AI agents.
Evaluate, select, and operationalize LLM frameworks and orchestration platforms (e.g., LangChain, LangGraph, CrewAI, n8n, or custom solutions) for production-grade agentic systems.
Implement guardrails, evaluation harnesses, and feedback loops to ensure AI agent outputs are accurate, safe, and continuously improving.
Champion the adoption of AI-assisted development and operations practices across the SRE and broader engineering organization.
Support the operations team’s incident response efforts, conduct post-mortems, and identify root causes to prevent recurrence.
Leverage AI tools to accelerate incident timelines, auto-generate post-mortem drafts, and surface patterns across historical incidents.
Document and share lessons learned, contributing to a culture of continuous improvement.
Identify repetitive operational workflows and engineer AI-augmented or fully automated replacements.
Build self-service tools and chatbot interfaces that allow engineering teams to query system status, retrieve logs, and execute standard operating procedures through natural language.
Measure and report on toil reduction metrics to quantify the impact of automation initiatives.
Work closely with developers, architects, and data/ML engineers to design solutions that improve reliability and leverage AI capabilities.
Collaborate with DevOps and NOC teams to support the application platform.
Communicate SRE practices, AI/automation capabilities, and operational insights to technical and non-technical stakeholders.
Provide feedback on application performance, potential improvements, and observability metrics.