AI Observability Benchmark for Kubernetes

KubeCon EU 2026: "The Observability Matrix" - Comparing AI-Driven Troubleshooting Tools

This repository contains all technical assets for benchmarking AI-driven observability tools for Kubernetes troubleshooting: HolmesGPT, Kagent, and Sympozium.

Overview

The benchmark evaluates each tool's ability to diagnose common Kubernetes failure patterns using a self-hosted LLM on Proxmox and Cluster API-managed kubeadm clusters. Scenarios range from basic infrastructure failures to advanced application-level issues using the OpenTelemetry Demo application with Gateway API routing (HTTPRoute/GRPCRoute), flagd feature flags, and kgateway as the Gateway API implementation.

Prerequisites

• Proxmox VE cluster with sufficient resources for LLM hosting
• Kind installed for management cluster
• kubectl, helm, clusterctl installed
• Minimum 64GB RAM and GPU (optional) for LLM
• Separate machine with OLLAMA reachable from your local network
• Anthropic API Key
• Dynatrace tenant (trial or existing)

Quick Start

1. Provision the Cluster

# Initialize Cluster API with Proxmox provider
clusterctl init --infrastructure proxmox

# Create the cluster
kubectl apply -f infrastructure/cluster-api/workload-cluster/capi_llm_cluster.yaml

# Wait for cluster to be ready
kubectl get clusters -w

# Get kubeconfig
cp ~/.kube/config ~/.kube/config-bck
clusterctl get kubeconfig observable-llm > observable-llm.kubeconfig
export KUBECONFIG=~/.kube/config:./observable-llm.kubeconfig
kubectl config view
kubectl config view --flatten > one-config.yaml
mv one-config.yaml ~/.kube/config
NAME=observable-llm

2. Install CNI

In order to get the nodes ready, you need a CNI installed change kubectl to point to our new cluster created by the cluster api

kubectl create ns kube-flannel
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged

helm repo add flannel https://flannel-io.github.io/flannel/
helm install flannel --set podCidr="192.168.0.0/16" --namespace kube-flannel flannel/flannel


kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl apply -f - -n kube-system

kubectl rollout restart ds kube-proxy -n kube-system

3. Environment Variables

Please replace the following variables with your details:

ANTHROPIC_API_KEY=<YOUR API KEY>
OLLAMA_LOCAL_URL=<http://YOUR OLLAMA URL>
CLUSTERNAME=<YOUR CLUSTER NAME>

4. Dynatrace Setup

4a. Dynatrace Tenant - Start a Trial

If you don't have any Dynatrace tenant, then I suggest to create a trial using the following link: Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example: https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT HOST>

4b. Create the Dynatrace API Tokens

The Dynatrace operator requires several tokens:

Operator Token — scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

API_TOKEN=<YOUR TOKEN VALUE>

Ingest Data Token — scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

4c. Create the Dynatrace Platform Token (for Dynatrace MCP Server)

The Dynatrace MCP server requires a Platform Token to query your environment via DQL, Davis Copilot, and entity APIs. This token is used by all three AI tools (HolmesGPT, Kagent, Sympozium) through the shared Dynatrace MCP server.

1. In your Dynatrace tenant, go to Account Management > Identity & Access Management > OAuth clients (or use the token management UI)
2. Create a new Platform Token (token format: dt0s16.SAMPLE.abcd1234)
3. Add the following scopes:

Scope	Purpose
davis-copilot:nl2dql:execute	Natural language to DQL
davis-copilot:dql2nl:execute	DQL to natural language
davis-copilot:conversations:execute	Davis Copilot conversations
storage:logs:read	Read logs via DQL
storage:events:read	Read events via DQL
storage:metrics:read	Read metrics via DQL
storage:spans:read	Read distributed traces
storage:entities:read	Read monitored entities
storage:buckets:read	Read storage buckets
storage:bizevents:read	Read business events
storage:security.events:read	Read security events
storage:smartscape:read	Read Smartscape topology
storage:system:read	Read system data

Important: Use your Dynatrace Platform URL (e.g., https://abc12345.apps.dynatrace.com), NOT the classic environment URL (*.live.dynatrace.com).

DT_PLATFORM_TOKEN=<YOUR PLATFORM TOKEN>

The deployment script creates the required Kubernetes secret:

kubectl create secret generic dynatrace-mcp -n dynatrace-mcp \
  --from-literal=DT_PLATFORM_TOKEN=$DT_PLATFORM_TOKEN \
  --from-literal=DT_ENVIRONMENT=$DT_TENANT_URL

For more details, see the Dynatrace MCP Server documentation.

4d. Create the Sympozium Slack App (Optional — for Slack channel integration)

Sympozium can interact with users via a Slack channel using Socket Mode. This is optional for the benchmark (the Web UI works without Slack), but useful for interactive troubleshooting demos.

1. Go to api.slack.com/apps and click Create New App > From scratch
2. Name the app (e.g., Sympozium SRE) and select your workspace
3. Enable Socket Mode under Settings > Socket Mode — this generates an App-Level Token (starts with xapp-)
   • Add the scope connections:write when creating the token
4. Under OAuth & Permissions, add these Bot Token Scopes:
   • app_mentions:read — react when mentioned
   • channels:history — read channel messages
   • channels:read — list channels
   • chat:write — send messages
   • im:history — read DMs
   • im:read — list DMs
   • im:write — send DMs
5. Install the app to your workspace — copy the Bot User OAuth Token (starts with xoxb-)
6. Invite the bot to the desired Slack channel: /invite @Sympozium SRE

Create the Kubernetes secret in the default namespace:

kubectl create secret generic sympozium-slack \
  --from-literal=SLACK_BOT_TOKEN=xoxb-your-bot-token \
  --from-literal=SLACK_APP_TOKEN=xapp-your-app-token

5. Deploy Everything

The deployment.sh script installs all infrastructure, observability stack, AI tools, and MCP servers:

./deployment.sh \
  --clustername "$CLUSTERNAME" \
  --dtoperatortoken "$API_TOKEN" \
  --dtingesttoken "$DATA_INGEST_TOKEN" \
  --dturl "$DT_TENANT_URL" \
  --platformtoken "$DT_PLATFORM_TOKEN" \
  --ollamaurl "$OLLAMA_LOCAL_URL" \
  --anthropickey "$ANTHROPIC_API_KEY"

This deploys:

• Gateway API CRDs + kgateway
• Istio (for GAMMA mesh routing)
• Cert-manager + OpenTelemetry Operator
• Dynatrace Operator + dynakube
• OTel Collector (DaemonSet + StatefulSet)
• OpenTelemetry Demo application
• MCP servers (mcp-k8s-networking, otel-collector-mcp, dynatrace-mcp)
• Sympozium (controller, API server, SkillPacks, MCP bridge)
• HolmesGPT (Helm chart, chat UI, MCP proxy sidecars)
• Kagent (CLI install, models, agents)

KPIs Measured

KPI	Description	Unit
Diagnosis Accuracy	Correctness of root cause identification	Score (0-3)
Token Consumption	LLM tokens used per diagnosis	Tokens
Resource Usage	CPU/Memory during diagnosis	Millicores/MiB
Time to Root Cause	Elapsed time for diagnosis	Seconds
User Interactions	Number of user inputs required	Count
Integrations	Available out-of-box integrations	Count

Scenarios

Tier 1: Basic Infrastructure Failures (01-06)

Standalone synthetic deployments testing fundamental K8s troubleshooting.

1. CrashLoopBackOff - Pod restart loops due to application errors
2. ImagePullBackOff - Container image pull failures
3. OOMKilled - Memory limit exceeded
4. Pending Pods - Scheduling failures
5. Service Connectivity - Service-to-pod communication issues
6. Gateway API Misconfiguration - HTTPRoute wrong backendRef (kgateway)

Tier 2: Gateway API Routing Failures (07-10)

Use the OTel Demo app with Gateway API (HTTPRoute/GRPCRoute) and kgateway. Routes use parentRefs with kind: Service for east-west mesh traffic (GAMMA initiative).

7. GRPCRoute backendRef Port Mismatch - GRPCRoute for payment service has backendRef port 9090 instead of 8080
8. HTTPRoute Path Mismatch - HTTPRoute for email service uses Exact path match missing /api prefix, silently returning 404
9. GRPCRoute Restrictive Method Match - GRPCRoute on product-catalog only matches ListProducts, rejecting GetProduct/SearchProducts
10. HTTPRoute Canary Weight Split Failure - HTTPRoute for frontend with 80/20 weight split where frontend-v2 backendRef doesn't exist

Tier 3: Application Failures via flagd Feature Flags (11-14)

Use OTel Demo's flagd feature flags to inject real application-level failures.

11. Cascading Checkout Failure - paymentFailure at 75% causes cascading checkout failures
12. Ad Service Degradation - adHighCpu + adManualGc compound to degrade the ad service
13. Kafka + Cart Dual Failure - kafkaQueueProblems + cartFailure create two independent failures
14. Product Catalog Cascade - productCatalogFailure + recommendationCacheFailure cascade through dependencies

Tools Compared

Tool	Type	LLM Support	CNCF Status
HolmesGPT	Root Cause Agent	OpenAI, Azure, Bedrock, Ollama, Anthropic	Sandbox
Kagent	Agent Framework	OpenAI, Azure, Anthropic, Ollama	Sandbox
Sympozium	Agentic Control Plane	Any LLM via SkillPacks	Early Stage

Recommended LLM

For self-hosted deployment compatible with all tools:

• Primary: qwen3.5:35b, qwen3.5:122b, gpt-oss:120b On your Ollama instance, pull all the models:

ollama pull qwen3.5:35b
ollama pull qwen3.5:122b
ollama pull gpt-oss:120b

Running the Benchmark

The benchmark is run scenario-by-scenario, testing each AI tool with the same prompt, then repeating across 4 LLM configurations.

Step 1: Access the Tool UIs

Open three terminal tabs to expose each tool's interface:

# Terminal 1 — Kagent Dashboard
kubectl port-forward svc/kagent-ui -n kagent 3000:3000
# Open: http://localhost:3000

# Terminal 2 — HolmesGPT Chat UI
kubectl port-forward svc/holmesgpt-chat-ui 8500:80 -n holmesgpt
# Open: http://localhost:8500

# Terminal 3 — Sympozium Web UI
kubectl port-forward svc/sympozium-apiserver -n sympozium-system 8080:8080
# Open: http://localhost:8080
# Get the UI token: kubectl get secret sympozium-ui-token -n sympozium-system -o jsonpath='{.data.token}' | base64 -d

Step 2: Run a Scenario

For each scenario, follow this loop:

# 1. Deploy the scenario
kubectl apply -f scenarios/<NN>-<name>/manifests.yaml

# For flagd scenarios (11-14), also restart flagd:
kubectl rollout restart deployment/flagd -n otel-demo
kubectl rollout status deployment/flagd -n otel-demo --timeout=300s

# 2. Open expected-diagnosis.md and copy the Test Prompt

# 3. Submit the SAME prompt to each tool:
#    - Kagent:    paste in the Kagent dashboard chat
#    - HolmesGPT: paste in the HolmesGPT chat UI
#    - Sympozium: paste in the Sympozium web UI (or Slack channel)

# 4. For each tool, record:
#    - Start time and end time (time to root cause)
#    - Number of chat interactions needed
#    - Score the response using the Scoring Criteria table
#    - Save the full conversation for later review

# 5. Clean up the scenario before moving to the next one
kubectl apply -f scenarios/opentelemetry-demo/manifest.yaml -n otel-demo

# For flagd scenarios, restart flagd to reset flags:
kubectl rollout restart deployment/flagd -n otel-demo

Step 3: Repeat with Each LLM

The benchmark tests 4 LLM configurations. After completing all 14 scenarios with one model, switch to the next and re-run everything.

Round 1 — gpt-oss:120b (default, already configured by deployment.sh):

# Run all 14 scenarios as described above

Round 2 — qwen3.5:35b:

# Update Kagent model config
kubectl apply -f tools/kagent/agents/qwen35/benchmark-agent.yaml
kubectl apply -f tools/kagent/agents/qwen35/benchmark-agent.yaml

# Update HolmesGPT
helm upgrade holmes oci://ghcr.io/henrikrexed/holmes -n holmesgpt \
  -f tools/holmesgpt/values_qwen3.5:35.yaml

# Update Sympozium agent
kubectl apply -f tools/sympozium/agents/qwen35/sympozium-sre-agent.yaml

# Re-run all 14 scenarios

Round 3 — qwen3.5:122b:

# Update Kagent model config
kubectl apply -f tools/kagent/agents/qwen122/benchmark-local.yaml
kubectl apply -f tools/kagent/agents/qwen122/benchmark-agent.yaml

# Update HolmesGPT
helm upgrade holmes oci://ghcr.io/henrikrexed/holmes -n holmesgpt \
  -f tools/holmesgpt/values_qwen3.5:122.yaml

# Update Sympozium agent
kubectl apply -f tools/sympozium/agents/qwen122/sympozium-sre-agent.yaml

# Re-run all 14 scenarios

Round 4 — Anthropic Claude (API):

# Update Kagent to use Claude
kubectl apply -f tools/kagent/agents/anthropic/benchmark-local.yaml
kubectl apply -f tools/kagent/agents/anthropic/benchmark-agent.yaml


# Update HolmesGPT
helm upgrade holmes oci://ghcr.io/henrikrexed/holmes -n holmesgpt \
  -f tools/holmesgpt/values_antrhopic.yaml

# Update Sympozium agent
kubectl apply -f tools/sympozium/agents/anthropic/sympozium-sre-agent.yaml

# Re-run all 14 scenarios

Step 4: Analyze Results

Resource consumption (CPU, memory, token usage) is collected automatically by Dynatrace via the OTel Collector and LiteLLM OTEL callbacks. Build a Dynatrace dashboard comparing each tool across all LLM configurations.

Results to compare per scenario:

• Diagnosis accuracy score (from expected-diagnosis.md scoring criteria)
• Time to root cause (start → answer)
• Number of user interactions
• Token consumption (from Dynatrace gen_ai.usage.* metrics)
• CPU/Memory usage (from Dynatrace container metrics)

Contributing

Contributions welcome! Please submit issues and PRs for:

• Additional scenarios
• Tool configuration improvements
• KPI measurement enhancements

License

Apache 2.0

References

• HolmesGPT
• Kagent
• Sympozium
• OpenTelemetry Demo
• flagd
• kgateway
• Gateway API
• Gateway API Mesh (GAMMA)
• Dynatrace MCP Server