Senior Software Engineer - Databases, SRE | Canada | Remote
About The Position
Grafana Labs is seeking a Senior Software Engineer - SRE to support high-value customers by enhancing the reliability of their Cloud databases, which are based on Mimir, Loki, Tempo, and Pyroscope. These databases are offered as a SaaS product across AWS, GCP, and Azure. The SRE team is integrated within the Mimir and Loki squads, focusing on delivering exceptional reliability for customers with high SLAs. This role requires operating at the intersection of customer needs, production systems, and product engineering. The company is 100% remote, with a global team and a culture rooted in open source, collaboration, and transparency. They encourage the use of modern AI coding assistants and offer competitive compensation and benefits, including Restricted Stock Units (RSUs).
Requirements
- 6+ years engineering experience, 3+ in SRE/CRE/production engineering
- Strong preference for those with formal customer reliability engineering experience
- Strong Kubernetes experience in AWS, GCP, or Azure
- Familiarity with infrastructure-as-code tooling (Helm, Terraform, Jsonnet, etc.)
- Experience operating multi-tenant systems in production
- Strong experience designing and implementing SLOs
- Experience with one or more programming languages (e.g. Go, Python, Java, etc)
- Experience with Linux operating systems internals
- Some knowledge of networking, cloud storage, and scaling
- Excellent problem-solving and troubleshooting skills
- Experience with calmly and actively participating in blame-free Incident Response, following up on actions, and writing high quality PIRs
- Ability to reason about performance, scaling, and failure modes
- Comfortable working within an engineering team where individuals are encouraged to have a strong sense of autonomy and self-direction
- Ability to partner deeply with product engineering teams
- Intellectually curious
- Default to transparency
- High bias towards action
- Kindness
Responsibilities
- Partner closely with product engineering squads (embedded model)
- Own production reliability for high-SLA and complex customer environments
- Design and implement automation to scale reliability practices
- Ensuring customers meet SLO targets
- Define and evolve per-tenant SLOs and reliability models
- Proactively reduce SLO burn to prevent repeat incidents
- Serve as a primary escalation point and on-call for relevant incidents
- Lead customer-impacting incident response and post-incident reviews
- Contribute to design docs and code reviews
- Influence feature design to ensure production scalability and operability
- Build automation to eliminate toil
- Improve alert quality and reduce noisy escalations
- Participate in on-call rotations, working with global counterparts for balanced coverage
- Reviewing and creating SLOs, investigating ways to reduce budget burn
- Improve observability of customers within their environments
- Design and implement solutions to ensure reliability and scalability
- Develop fault-tolerant design patterns
- Collaborate with Engineering Leaders to define and influence product strategy, roadmaps, and technical designs
- Participate in PR review and collaborate on Design Docs
- Teach others about Site Reliability Engineering and communicate best practices
- Participate in Incident Response, including investigation, resolution, PIR, and customer communication
Benefits
- Restricted Stock Units (RSUs)
- 30 days of annual leave
- 3 Grafana Shutdown Days
- Company-funded usage budget for AI coding assistants
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
