Staff Backend Engineer - Second Horizon | US | Remote

Grafana Labs

8d•$174,986 - $209,983•Remote

About The Position

Grafana Labs is seeking a Staff Backend Engineer to join a new skunkworks initiative focused on bringing observability to the rest of the business. The goal is to make Grafana the central platform for understanding and acting on real-time enterprise data, powered by AI. This role involves building an AI-native data intelligence system that provides agents with governed access to enterprise context, including data, metadata, lineage, quality signals, and institutional knowledge. The team operates with high autonomy and ownership, empowering engineers to make decisions and iterate quickly within a collaborative culture. This is an early-stage role on a high-autonomy team, requiring comfort with ambiguity, pragmatic architectural decisions, and building scalable systems from prototype to production-grade SaaS. There is significant opportunity for growth and to redefine the role based on impact and initiative.

Requirements

Strong engineering skills with solid experience building production-grade, user-facing software systems.
Self-starter capable of tackling complex engineering problems and making UI design decisions with minimal supervision.
Familiarity with AI technologies and frameworks, with a focus on delivering high-quality, real-world solutions.
Comfortable releasing prototypes, collecting feedback, and iterating with a pragmatic mindset.
Proven initiative, ownership, and ability to drive projects forward, pushing boundaries for impactful solutions.
Ability to deal with ambiguity and define scope where things are loosely defined.
Effective communication and a collaborative attitude, open to feedback and bringing a solutions-oriented mindset.
Experience with LLMs, prompt engineering, and building applications powered by GenAI.
Proven track record of delivering software that made it into production and is actively used by users.
Exposure to working in cloud-native environments (e.g., AWS, GCP, Azure).
Experience using observability tools to understand and troubleshoot system behavior.

Nice To Haves

Experience building or working with agent frameworks or multi-agent workflows.
Experience as a data analyst or work with data platforms (e.g., Looker, Tableau, PowerBI, Snowflake, DataBricks).
Experience building tools for data engineering.

Responsibilities

Design, implement, test, and operate the first services for context ingestion, context indexing, retrieval orchestration, API access, source configuration, and system administration.
Help define and build the architecture for a multi-tenant service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
Build APIs and service interfaces that allow AI agents, MCP tools, CLIs, and internal applications to retrieve relevant context, provenance, confidence signals, and warnings.
Partner with the team to make practical tradeoffs between fast experimentation and long-term reliability, especially as the project moves from prototype to production.
Instrument services with metrics, logs, traces, alerts, and dashboards. Use observability tools to understand system behavior and improve reliability.
Help shape the architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices for a new product area.
Communicate effectively and contribute across teams in a dynamic and collaborative environment.
Take full ownership of the AI solutions developed, ensuring they are innovative, scalable, maintainable, and aligned with real user workflows.