Staff AI Engineer | Canada | Remote

Grafana Labs

Contract type

Ongoing

Work mode

Remote · Canada

Experience

Senior · 8+ years

Job description

Staff Engineer (AI & Automation) at Grafana Labs who owns the AI agent infrastructure and automation platform powering the Marketing Operations organization. The role builds multi-agent architectures, LLM integrations, and backend services connecting AI models to internal and third-party data platforms.

At a glance

Employment: Full-time · Employee (employment agreement)
Work mode: Remote · Canada
Pay: CAD 164,490 – 197,389 / year

What you'll do

Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools)
Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure
Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope high-impact automation problems, identify bottlenecks, and build solutions with measurable business outcomes
Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently

What we're looking for

8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering
2+ years hands-on experience applying LLMs/AI to production workflows, not just prototypes
Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline
Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation
Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns (sequential chains, router/dispatcher, parallel fan-out), state management, and production monitoring
Diagnose business problems before writing code; think in workflows and outcomes, not just functions
Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction
Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code)
Clear technical communicator who can explain complex systems in simple terms to both engineers and business stakeholders

Nice to have

Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, Qdrant, pgvector)
Familiarity with marketing or sales platforms (Salesforce, Customer.io, HubSpot, Marketo, Outreach)
Experience with frontend frameworks (React, Slack Block Kit) for building user-facing AI tool interfaces
Observability tooling for AI systems (LangSmith, Weights & Biases, custom evaluation frameworks)
Experience with workflow orchestration platforms (n8n, Temporal, Prefect, Airflow)
Familiarity with Model Context Protocol (MCP) or similar standards for connecting AI systems to data sources
Prior work automating marketing, sales, or customer success workflows in a B2B SaaS environment
Active in open-source communities

Your team

Marketing Operations

The Marketing Operations organization, partnering with RevOps, Demand Generation, Regional Marketing, and SDR teams

Benefits

100% Remote, Global Culture
Scaling Organization
Transparent Communication
Innovation-Driven
Open Source Roots
Empowered Teams
Career Growth Pathways
Approachable Leadership
Passionate People
In-Person onboarding
Global annual leave policy of 30 days per annum, with 3 days reserved for Grafana Shutdown Days
Restricted Stock Units (RSUs)

Save

About Grafana Labs

The company behind Grafana Cloud, the fully managed observability platform

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards and designed for interoperability across any stack, Grafana Cloud brings AI to observability and observability to AI. We are a 100% remote company with team members across 40+ countries.

Key facts

Observability / software
100% remote company with team members across 40+ countries
Grafana Cloud trusted by more than 10,000 organizations
Backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P. Morgan, CapitalG, and Lead Edge Capital
Built on open source and open standards

Partners & customers

AnthropicBloombergNVIDIAMicrosoftSalesforce

What they value

Transparency
Autonomy
Trust
Collaboration
Open source roots
High trust, low ego culture that values outcomes over optics