Staff Backend Engineer - Second Horizon | Canada | Remote

Grafana Labs

8d•CA$186,368 - CA$223,642•Remote

About The Position

Grafana Labs is seeking a Staff Backend Engineer to join a new skunkworks initiative focused on bringing observability to the rest of the business. The goal is to create an AI-native data intelligence system that provides AI agents with reliable, governed access to enterprise context, including data, metadata, definitions, lineage, quality signals, and institutional knowledge. This system aims to be the single best place for humans and AI agents to understand and act on real-time enterprise data. The role involves designing and shipping the core backend architecture for ingestion, context storage, retrieval APIs, and agent-facing integrations. This is an early-stage, high-autonomy role where the engineer will be comfortable working through ambiguity, making pragmatic architectural decisions, and building systems that can evolve from internal dogfooding to production-grade SaaS. As the team matures, there is an opportunity to expand or redefine the role based on impact and initiative.

Requirements

Strong engineering skills with solid experience building production-grade, user-facing software systems.
Self-starter capable of tackling complex engineering problems and making UI design decisions with minimal supervision.
Familiarity with AI technologies and frameworks, with a focus on delivering high-quality solutions that work in the real world.
Comfortable releasing prototypes, collecting feedback, and iterating with a pragmatic mindset.
Proven initiative, ownership, and ability to drive projects forward, pushing boundaries to find the most impactful solutions.
Ability to deal with ambiguity and define scope where things are loosely defined.
Collaborative attitude with effective communication with peers, openness to feedback, and a solutions-oriented mindset.
Experience with LLMs, prompt engineering, and building applications powered by GenAI.
Proven track record of delivering software that made it into production and is actively used by users.
Exposure to working in cloud-native environments (e.g., AWS, GCP, Azure).
Experience using observability tools to understand and troubleshoot system behavior.

Nice To Haves

Experience building or working with agent frameworks or multi-agent workflows.
Experience as a data analyst or work with data platforms (e.g., Looker, Tableau, PowerBI, Snowflake, DataBricks).
Experience building tools for data engineering.

Responsibilities

Design, implement, test, and operate the first services for context ingestion, context indexing, retrieval orchestration, API access, source configuration, and system administration.
Help define and build the architecture for a multi-tenant service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
Build APIs and service interfaces that allow AI agents, MCP tools, CLIs, and internal applications to retrieve relevant context, provenance, confidence signals, and warnings.
Partner with the team to make practical tradeoffs between fast experimentation and long-term reliability, especially as the project moves from prototype to production.
Instrument services with metrics, logs, traces, alerts, and dashboards. Use observability tools to understand system behavior and improve reliability.
Help shape the architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices for a new product area.
Communicate effectively and contribute across teams in a highly dynamic and collaborative environment.
Take full ownership of the AI solutions developed, ensuring they are innovative, scalable, maintainable, and aligned with real user workflows.