Staff Software Engineer - Grafana Cloud Observability, Kubernetes Monitoring | USA | Remote

Grafana Labs

5d•Remote

About The Position

Grafana Cloud is our composable observability platform that integrates metrics, logs, and traces with Grafana. It allows our customers to leverage the best open source observability software – including Prometheus, Mimir, Loki, and Tempo – without the overhead of installing, maintaining and scaling their own observability stack The Observability department is focused on enabling developers to understand the health and performance of their applications and infrastructure in any environment by providing tools to instrument their code, ingest observability data into Grafana Cloud and visualize and explore it. In this role, you will be part of the team that builds our Cloud Observability stack that allows customers to collect and visualize metrics from various systems and applications. We build and maintain the backend of opinionated applications such as Cloud Provider Observability, Database Observability, and Kubernetes Monitoring. This includes the dashboards, alerts, documentation, and infrastructure while working closely with other teams to ensure seamless experiences. We also strive to incorporate OSS contributions in our work by contributing to projects such as Alloy, Prometheus, OpenTelemetry, and Beyla. The Observability department provides a core building block for customers using Grafana Cloud. As a company we are remote-first and global, we embrace people of different experiences and backgrounds to build diverse teams where every person brings a unique perspective to the software. We are looking for Engineers that are passionate about communicating with data and providing seamless experiences for our customers to join our growing team! Engineers at Grafana also have the opportunity to contribute to Open Source communities. At Grafana Labs, our engineers have a dedicated career path and do not have to become managers to progress in their career. Staff Software Engineers at Grafana have a large amount of experience across multiple areas. They are able to estimate, plan, coordinate and deliver large tasks spanning multiple systems. They actively coach and mentor other team members in their team and are able to identify and resolve issues with technology and product processes. You will bring your passion for observability and software engineering expertise to help us take our infrastructure monitoring capabilities within Grafana Cloud to the next level. This will include working with our Kubernetes monitoring solution. As we are remote-first and our engineering organization is entirely remote, we provide guidance and meet regularly using video calls, so an independent attitude, good communication skills, and transparency are a must. We invest heavily in developer productivity. You can use modern AI coding assistants as part of your daily workflow (your choice of tools, within security guidelines), backed by a company-funded usage budget so you can iterate quickly without unnecessary friction. We encourage pragmatic AI-assisted development: faster prototyping, test generation, refactors, documentation, and incident follow-ups—always paired with strong code review and quality standards. You’ll also have access to frontier models (e.g., GPT-Codex 5/3, Claude Opus 4.6, Gemini 3 Pro).

Requirements

Strong 8+ years of experience with at least one programming language - any major language (Python, .NET, Java, Go, Rust, etc) is acceptable
Demonstrated working experience in operating high-scale production systems running on Kubernetes and monitoring it, including on-call participation, incident response, and postmortem practices
Familiarity with observability tooling (e.g. Grafana)
Strong understanding of time-series data, metrics cardinality challenges, and cost/performance tradeoffs/optimizations in observability systems
Experience in a hands-on technical leadership role - setting technical direction, leading project teams, and influencing architectural decisions beyond your immediate team
Deep understanding of distributed systems concepts including scalability, consistency, high availability, and failure modes in large-scale systems
Experience writing clean, maintainable, robust, and performant software
Experience with delivering projects from start to finish in a self-driven manner
Excellent problem-solving and debugging skills
Strong mentoring and leadership skills

Nice To Haves

Experience operating or scaling Prometheus in high-cardinality, multi-tenant environments
Experience working with OpenTelemetry Collector pipelines or similar telemetry ingestion systems
Certified Kubernetes Administrator (CKA)/ Certified Kubernetes Application Developer (CKAD) or any other Kubernetes related certification from CNCF
Experience developing Kubernetes operators, controllers, or custom resources
Strong understanding of metrics collection, visualization, and alerting concepts
Experience contributing to or maintaining open source projects, with evidence of successful pull requests and community collaboration
Experience designing and building observability backends for various systems and applications

Responsibilities

Design and implement high-quality, scalable integrations for various infrastructure components, applications, and data ingestion pipelines
Create middleware components and libraries that simplify development and maintenance of observability solutions
When necessary, represent Grafana Labs in open source forums, working groups, and events
Work with product teams, in addition to design and docs, to develop features that align with wider product strategy and customer needs
Lead the technical direction and vision of the team, contributing to strategic discussions and future development of observability solutions
Work with other departments including Sales, Product, and Support teams to deliver a holistic product experience
Take ownership of the services you’re running by deploying well tested clean code
Embrace our open-source culture and contribute to other projects that may not directly fall within your team’s scope

Benefits

All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success.
We believe in shared outcomes—RSUs help us stay aligned and invested as we scale globally.
100% Remote, Global Culture - As a remote-only company, we bring together talent from around the world, united by a culture of collaboration and shared purpose.
Scaling Organization – Tackle meaningful work in a high-growth, ever-evolving environment.
Transparent Communication – Expect open decision-making and regular company-wide updates.
Innovation-Driven – Autonomy and support to ship great work and try new things.
Open Source Roots – Built on community-driven values that shape how we work.
Empowered Teams – High trust, low ego culture that values outcomes over optics.
Career Growth Pathways – Defined opportunities to grow and develop your career.
Approachable Leadership – Transparent execs who are involved, visible, and human.
Passionate People – Join a team of smart, supportive folks who care deeply about what they do.
In-Person onboarding - We want you to thrive from day 1 with your fellow new ‘Grafanistas’ to learn all about what we do and how we do it.
Balance is Key - We operate a global annual leave policy of 30 days per annum. 3 days of your annual leave entitlement are reserved for Grafana Shutdown Days to allow the team to really disconnect. We will comply with local legislation where applicable.