Staff AI Engineer | US | Remote

Grafana Labs

3d • $154,445 - $185,334 • Remote

About The Position

Grafana Labs is seeking a Staff Engineer (AI & Automation) to own the AI agent infrastructure and automation platform supporting its Marketing Operations organization. The role builds multi-agent architectures, LLM integrations, and backend services connecting AI models with internal and third-party data platforms, and delivers production systems used by teams daily.

This is a high-autonomy position with ownership of technical direction. The engineer will identify high-leverage problems across Marketing, RevOps, and SDR teams; design and deliver solutions; define the automation platform’s data models, API contracts, shared libraries, and reference architectures; and partner with Data Engineering, GTM Systems, and Field Operations on scalable, self-service automation.

The position is remote and open to candidates in the United States. Grafana Labs provides AI coding assistants for development, subject to security guidelines, and encourages pragmatic AI-assisted development with strong code review and quality standards.

Requirements

• At least 8 years of software engineering experience, with depth in backend development, systems integration, or data/analytics engineering.
• At least 2 years of hands-on experience applying LLMs or AI to production workflows, beyond prototypes.
• Strong proficiency in Python and JavaScript/Node.js.
• Experience with Git-based workflows, code review practices, and disciplined testing.
• Hands-on experience with LLM frameworks and patterns, including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation.
• Experience building and operating multi-agent systems at scale, including agent decomposition, orchestration patterns, state management, and production monitoring.
• Ability to diagnose business problems before writing code and focus on workflows and outcomes.
• Deep familiarity with Google Cloud Platform, BigQuery, and serverless or containerized services such as Cloud Functions and Cloud Run.
• Understanding of LLM failure modes and production mitigations, including confidence thresholds, fallback logic, human escalation, and cost and latency management.
• Ability to identify high-leverage problems, push back on low-impact requests, and deliver end to end with minimal direction.
• Fluency with AI-assisted development tools such as GitHub Copilot, Cursor, or Claude Code.
• Clear technical communication skills, including explaining complex systems to engineers and business stakeholders.

Nice To Haves

• Experience with vector databases or retrieval pipelines such as Pinecone, Weaviate, ChromaDB, Qdrant, or pgvector.
• Familiarity with marketing or sales platforms such as Salesforce, Customer.io, HubSpot, Marketo, or Outreach.
• Experience with frontend frameworks such as React or Slack Block Kit for building user-facing AI tool interfaces.
• Experience with AI observability tooling such as LangSmith, Weights & Biases, or custom evaluation frameworks.
• Experience with workflow orchestration platforms such as n8n, Temporal, Prefect, or Airflow.
• Familiarity with Model Context Protocol (MCP) or similar standards for connecting AI systems to data sources.
• Prior experience automating marketing, sales, or customer success workflows in a B2B SaaS environment.
• Active participation in open-source communities.

Responsibilities

• Own the end-to-end development of multi-agent AI systems, including architecture, implementation, testing, deployment, and ongoing operation.
• Build modular, composable agentic systems using orchestration frameworks such as LangChain, CrewAI, or Anthropic MCP, designed to operate across teams around the clock.
• Develop reusable agentic skills that can be invoked through Slack, dashboards, internal apps, and CLIs.
• Implement observability and feedback loops, including logging, performance metrics, prompt iteration, model evaluation, and cost management.
• Establish governance and compliance standards for AI workflows, including access controls, audit trails, PII handling, and human-in-the-loop escalation paths.
• Build MCP servers, APIs, CLIs, and microservices that connect AI models to business systems such as BigQuery, Slack, CRMs, email, calendars, and analytics tools.
• Architect RAG data flows connecting LLMs to internal knowledge bases, customer data, and real-time business context.
• Build serverless or containerized services using platforms such as GCP Cloud Functions and Cloud Run, scaling with usage and integrating with Grafana’s cloud infrastructure.
• Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to identify automation opportunities, bottlenecks, and measurable business outcomes.
• Design and deploy workflows using orchestration tools such as n8n, Workato, or custom platforms, with CI/CD, testing, and production reliability standards.
• Create self-service systems, documentation, playbooks, and enablement materials so partner teams can operate independently.
• Define the automation platform’s technical direction, including data models, API contracts, shared libraries, and reference architectures.
• Collaborate with Data Engineering, GTM Systems, and Field Operations to build scalable automation that reduces manual work and improves operational efficiency.

Benefits

• Restricted Stock Units (RSUs) for all roles.
• 100% remote work.
• Career growth pathways and opportunities to develop.
• In-person onboarding.
• Global annual leave policy of 30 days per annum, including 3 Grafana Shutdown Days, subject to local legislation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed