Skip to content

Commit 6502e54

Browse files
Feature/grafana irm plugin(#9127) (#9160)
* feat(grafana_irm): add Grafana IRM incident collection plugin * test(grafana_irm): add richer live-captured e2e/unit fixtures; implement TestConnection - Seed 2 more real (non-drill) incidents on the dev stack with a real role assignment and multi-label data, closing gaps the original 2-incident fixture never exercised (assignment mapping, per-scope label exclusion). - Add a second e2e scenario proving label-based scope filtering end-to-end. - Update the extractor unit test to use real captured incident data instead of a drill incident's. - Implement TestConnection/TestExistingConnection via a cheap IncidentsService.QueryIncidents call, verified live against the real API. * refactor(grafana_irm): revert to connection-wide scoping, drop label filtering The originating feature request only asks for org-wide DORA correlation and never mentions per-team scoping. The label-based per-scope filtering added earlier this session was real and tested, but solved a problem nobody asked for, and config-ui has no generic manual-scope-entry component to build a UI for it on top of. Revert GrafanaIrmScopeConfig/GrafanaIrmOptions/the collector/converter to a single connection-wide scope with no filtering, same shape as incidentio/pagerduty. All incident labels are still collected and stored unconditionally, just no longer used to gate conversion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(grafana_irm): add config-ui frontend and single-scope remote API Backend: add remote_api.go returning the one synthetic scope every connection has (config-ui's data-scope picker has no manual-entry flow to fall back to, so this feeds the standard remote-picker UI a single 'All Incidents' entry instead), wired into impl.go's routes. Frontend: add config-ui/src/plugins/register/grafana_irm (connection form, icon, registration) mirroring incidentio's shape, using the Grafana logo already vendored under grafana/img/. No custom scope-config UI is needed since GrafanaIrmScopeConfig carries no extra fields (Option A). Verified with tsc, eslint, and a production vite build - all clean. * fix(grafana_irm): fix error raised during testing * refactor(grafana_irm): collapse collect+refresh into one FinalizableApiCollector subtask The two-subtask design (CollectIncidents + RefreshOpenIncidents) caused three real bugs found via live pipeline testing: a panic, "Re-transform Data" silently deleting resolved incidents, and a genuine full sync also losing data because both subtasks independently wiped the raw table with no coordination between them. pagerduty solves the identical "list new/changed, then refresh still-open" need with DevLake's own FinalizableApiCollector as a single subtask, which prevents these bugs by construction (the framework never builds the "refresh" half on a full sync, and there's only one subtask name for Re-transform Data's filter to strip). Migrated CollectIncidents to that pattern; all existing collector logic (query building, change-detection via a request-scoped header) preserved, just re-wired. Live-verified against a real server: full sync restores all incidents correctly, incremental sync has no regression, and Re-transform Data no longer drops resolved incidents. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(grafana_irm): add dedicated Grafana dashboard, docs, and domain board mapping - Add dedicated Grafana dashboards for MySQL and PostgreSQL dialects - Populate domain ticket.Board in ConvertIncidents to enable board filtering and joins - Update config-ui brand icon with official Grafana flame SVG - Add plugin documentation in backend/plugins/grafana_irm/README.md - Fix struct alignment formatting in models/incident.go * fix(grafana_irm): improve connection test error messages and add docs link * fix(grafana_irm): truncate timestamps to milliseconds and fix e2e postgres scope setup --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent f3d8859 commit 6502e54

52 files changed

Lines changed: 7266 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
# Grafana IRM Plugin
19+
20+
## Summary
21+
22+
The `grafana_irm` plugin connects Apache DevLake to **Grafana Cloud Incident Response & Management (IRM)**. It ingests incident records, assignments, and labels into DevLake and transforms them into standardized domain models (`Issue`, `IssueAssignee`, `IssueLabel`, and `BoardIssue`) to calculate key DORA metrics:
23+
24+
- **Change Failure Rate (CFR)**
25+
- **Mean Time to Recovery / Failed Deployment Recovery Time (MTTR)**
26+
27+
This plugin follows the established incident-management pattern used by DevLake plugins like `pagerduty` and `incidentio`.
28+
29+
---
30+
31+
## What It Collects
32+
33+
### Source API Endpoints
34+
35+
Grafana IRM uses a JSON-RPC HTTP POST API under `/api/plugins/grafana-irm-app/resources/api/v1/`:
36+
37+
| RPC Method | Purpose |
38+
| :--- | :--- |
39+
| `IncidentsService.QueryIncidents` | Query incident list with pagination, date bounds, and drill filters |
40+
| `IncidentsService.GetIncident` | Fetch details for open / active incidents during the refresh pass |
41+
42+
### Database Tables
43+
44+
#### Tool Layer
45+
- `_tool_grafana_irm_connections`: Connection configurations
46+
- `_tool_grafana_irm_scopes`: Data scopes (connection-wide incident feed)
47+
- `_tool_grafana_irm_scope_configs`: Scope transformation configurations
48+
- `_tool_grafana_irm_incidents`: Incident metadata, severities, durations, and timestamps
49+
- `_tool_grafana_irm_incident_labels`: Key-value labels attached to incidents
50+
- `_tool_grafana_irm_incident_assignments`: Role-based incident responder assignments
51+
52+
#### Domain Layer
53+
- `issues`: Standardized issue entity (`type = 'INCIDENT'`)
54+
- `board_issues`: Mapping of incidents to the connection scope board
55+
- `issue_labels`: Incident labels (useful for team/service breakdown in SQL/Grafana)
56+
- `issue_assignees`: Incident assignees and roles (commander, investigator, etc.)
57+
58+
---
59+
60+
## Data Flow
61+
62+
```mermaid
63+
flowchart LR
64+
API["Grafana Cloud IRM API\n(JSON-RPC POST)"]
65+
RAW[("_raw_grafana_irm_incidents")]
66+
TOOL[("_tool_grafana_irm_incidents\n_tool_grafana_irm_incident_labels\n_tool_grafana_irm_incident_assignments")]
67+
DOMAIN[("Domain Layer\nissues\nissue_labels\nissue_assignees\nboard_issues")]
68+
DORA["DORA Dashboards\n(CFR, MTTR)"]
69+
70+
API -->|collectIncidents| RAW
71+
RAW -->|extractIncidents| TOOL
72+
TOOL -->|convertIncidents| DOMAIN
73+
DOMAIN --> DORA
74+
```
75+
76+
### Pipeline Subtasks
77+
78+
The plugin executes three subtasks in order:
79+
1. `collectIncidents`: Pulls new and updated incidents from Grafana Cloud using cursor pagination and re-fetches unresolved incidents.
80+
2. `extractIncidents`: Extracts raw JSON payloads into tool-layer tables.
81+
3. `convertIncidents`: Transforms tool records into standardized DevLake domain tables for DORA analysis.
82+
83+
---
84+
85+
## Collection Strategy & Sync Architecture
86+
87+
1. **Incremental Collection**:
88+
- Uses query DSL date ranges: `isdrill:false or(declared:<since>,<until> resolved:<since>,<until>)`.
89+
- Filters out drill/simulation incidents server-side to avoid polluting production DORA metrics.
90+
2. **Unified `FinalizableApiCollector`**:
91+
- Implements both new incident listing and open-incident refresh within a single `FinalizableApiCollector` subtask.
92+
- On full syncs, the refresh phase is bypassed by design to prevent raw table clobbering.
93+
- On "Re-transform Data", the entire collector is cleanly bypassed without dropping previously resolved incidents.
94+
3. **Smart Skip for Unchanged Incidents**:
95+
- When refreshing open incidents, the collector passes `X-Devlake-Known-Modified` headers.
96+
- If the remote incident has not changed since the last sync, insertion into the raw table is skipped to prevent table bloating.
97+
98+
---
99+
100+
## Setup & Configuration
101+
102+
### Prerequisites
103+
104+
1. An active **Grafana Cloud** stack with IRM (Incident Response & Management) enabled.
105+
2. A **Grafana Cloud Service Account Token** with permissions to read incidents (`Incident: Read` or viewer/editor role on the IRM app).
106+
> **Note:** On brand-new stacks, open the IRM app in your Grafana Cloud web UI at least once before creating API connections to trigger internal org provisioning.
107+
108+
### Step 1 — Create Connection
109+
110+
1. In DevLake Config UI (`http://localhost:4000`), navigate to **Connections** → **Add Connection** → select **Grafana IRM**.
111+
2. Configure the following fields:
112+
- **Connection Name**: A memorable name for your stack (e.g. `Production Grafana IRM`).
113+
- **Grafana Cloud Stack URL**: The base URL of your Grafana Cloud instance (e.g. `https://mycompany.grafana.net/`).
114+
- **Service Account Token**: The bearer token generated from Grafana Cloud (e.g. `glsa_...`).
115+
- **Rate Limit**: Optional hourly request limit (defaults to `3,600` requests/hour).
116+
3. Click **Test Connection**. Once verified, click **Save Connection**.
117+
118+
### Step 2 — Add Data Scope
119+
120+
1. Click **Add Data Scope** for the saved connection.
121+
2. Select the **All Incidents** scope checkbox.
122+
- Grafana IRM manages incidents globally across the stack; selecting "All Incidents" ingests the organization's incident stream.
123+
3. Click **Save Scope**.
124+
125+
### Step 3 — Collect Data in a Project
126+
127+
1. Navigate to **Projects** from the sidebar and open your target project (or create a new one).
128+
2. Under **Project Metrics**, ensure **DORA Metrics** is enabled.
129+
3. Click **Add Connection** and select your Grafana IRM connection with the **All Incidents** scope.
130+
4. Configure your sync frequency and historical date range, then click **Save**.
131+
5. Trigger or wait for the blueprint pipeline run to complete.
132+
133+
---
134+
135+
## Domain Mapping Reference
136+
137+
| Grafana IRM Field | DevLake Domain Field (`issues`) | Notes |
138+
| :--- | :--- | :--- |
139+
| `incidentID` | `original_id` | Unique incident ID |
140+
| `title` | `title` | Incident title / summary |
141+
| `status` | `status`, `original_status` | Mapped to `DONE` (`resolved`) or `IN_PROGRESS` (`active`) |
142+
| `severity` | `severity`, `priority` | Critical, Major, Minor |
143+
| `createdTime` / `incidentStart` | `created_date` | Incident declaration time |
144+
| `closedTime` / `incidentEnd` | `resolution_date` | Incident resolution timestamp (empty if unresolved) |
145+
| `durationSeconds` | `lead_time_minutes` | Converted to minutes for MTTR calculations |
146+
| `labels[]` | `issue_labels` (join table) | Ingested as key-value label rows (`label_key:label`) |
147+
| `assignments[]` | `issue_assignees` (join table) | User assignments and roles (e.g. `commander`) |
148+
| `isDrill` | *Filtered out* | Filtered server-side (`isdrill:false`) |
149+
150+
---
151+
152+
## Troubleshooting
153+
154+
| Issue | Cause & Resolution |
155+
| :--- | :--- |
156+
| **401 Unauthorized** on Test Connection | Invalid token, expired token, or token lacks IRM access. Verify service account permissions in Grafana Cloud. |
157+
| **User not found / Org not found** | Brand new stack where IRM was never opened. Log in to Grafana Cloud and click into the IRM application once to complete initial org setup. |
158+
| **Rate Limit (429)** | Exceeded Grafana Cloud API limits. Adjust `rateLimitPerHour` in connection settings or wait for rate limit window reset. |
159+
| **Missing incidents in DORA metrics** | Check that deployments are configured in DevLake so incidents can be correlated with deployments. Also verify whether the incidents were drills (`isDrill: true`), which are intentionally excluded. |
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"github.com/apache/devlake/core/errors"
22+
coreModels "github.com/apache/devlake/core/models"
23+
"github.com/apache/devlake/core/models/domainlayer/didgen"
24+
"github.com/apache/devlake/core/models/domainlayer/ticket"
25+
"github.com/apache/devlake/core/plugin"
26+
"github.com/apache/devlake/core/utils"
27+
"github.com/apache/devlake/helpers/pluginhelper/api"
28+
"github.com/apache/devlake/helpers/srvhelper"
29+
"github.com/apache/devlake/plugins/grafana_irm/models"
30+
"github.com/apache/devlake/plugins/grafana_irm/tasks"
31+
)
32+
33+
func MakeDataSourcePipelinePlanV200(
34+
subtaskMetas []plugin.SubTaskMeta,
35+
connectionId uint64,
36+
bpScopes []*coreModels.BlueprintScope,
37+
) (coreModels.PipelinePlan, []plugin.Scope, errors.Error) {
38+
connection, err := dsHelper.ConnSrv.FindByPk(connectionId)
39+
if err != nil {
40+
return nil, nil, err
41+
}
42+
scopeDetails, err := dsHelper.ScopeSrv.MapScopeDetails(connectionId, bpScopes)
43+
if err != nil {
44+
return nil, nil, err
45+
}
46+
plan, err := makePipelinePlanV200(subtaskMetas, scopeDetails, connection)
47+
if err != nil {
48+
return nil, nil, err
49+
}
50+
scopes, err := makeScopesV200(scopeDetails, connection)
51+
return plan, scopes, err
52+
}
53+
54+
func makePipelinePlanV200(
55+
subtaskMetas []plugin.SubTaskMeta,
56+
scopeDetails []*srvhelper.ScopeDetail[models.GrafanaIrmScope, models.GrafanaIrmScopeConfig],
57+
connection *models.GrafanaIrmConnection,
58+
) (coreModels.PipelinePlan, errors.Error) {
59+
plan := make(coreModels.PipelinePlan, len(scopeDetails))
60+
for i, scopeDetail := range scopeDetails {
61+
stage := plan[i]
62+
if stage == nil {
63+
stage = coreModels.PipelineStage{}
64+
}
65+
66+
scope, scopeConfig := scopeDetail.Scope, scopeDetail.ScopeConfig
67+
task, err := api.MakePipelinePlanTask(
68+
"grafana_irm",
69+
subtaskMetas,
70+
scopeConfig.Entities,
71+
tasks.GrafanaIrmOptions{
72+
ConnectionId: connection.ID,
73+
ScopeId: scope.Id,
74+
},
75+
)
76+
if err != nil {
77+
return nil, err
78+
}
79+
stage = append(stage, task)
80+
plan[i] = stage
81+
}
82+
83+
return plan, nil
84+
}
85+
86+
func makeScopesV200(
87+
scopeDetails []*srvhelper.ScopeDetail[models.GrafanaIrmScope, models.GrafanaIrmScopeConfig],
88+
connection *models.GrafanaIrmConnection,
89+
) ([]plugin.Scope, errors.Error) {
90+
scopes := make([]plugin.Scope, 0, len(scopeDetails))
91+
92+
idgen := didgen.NewDomainIdGenerator(&models.GrafanaIrmScope{})
93+
for _, scopeDetail := range scopeDetails {
94+
scope, scopeConfig := scopeDetail.Scope, scopeDetail.ScopeConfig
95+
id := idgen.Generate(connection.ID, scope.Id)
96+
97+
if utils.StringsContains(scopeConfig.Entities, plugin.DOMAIN_TYPE_TICKET) {
98+
scopes = append(scopes, ticket.NewBoard(id, scope.Name))
99+
}
100+
}
101+
102+
return scopes, nil
103+
}

0 commit comments

Comments
 (0)