Investigation: Approved SPs fraction failing by check type alerts

[silent-cipher]

Description

The "Approved SPs fraction failing by check type (sev2.5: >= 50% of SPs failing)" chart on the Dealbot Operational Dashboard fires frequently and auto-resolves shortly after, without a corresponding real SP-side outage.

• Chart: Approved SPs fraction failing by check type (id 12253190876)
• Dashboard: Dealbot Operational Dashboard (id 618457), source Infra Prod, mainnet only
• Backed by 4 separate threshold alerts, one per check type, all >= 50% of approved SPs failing:
  • 2241582239 — dataStorageStatus
  • 2241582240 — retrievalStatus
  • 2241582241 — dataSetChallengeStatus
  • 2715738984 — sampledPieceRetrievalStatus
• All 4 share the same config: confirmation_period=300s, recovery_period=1800s, notification=email, escalation=current_team.
• The 30-minute recovery period matching the frequent auto-resolve pattern is itself a hint these are transient blips, not sustained failures.

Investigation so far

One instance of this pattern (alert 2715738984, sampledPieceRetrievalStatus) was investigated on 2026-07-09 (~12:45–13:35 UTC) and confirmed as a false positive, not a real retrieval outage:

1. Only 2–3 approved mainnet SPs generate volume for that check type — the fleet-level query only requires active_sps >= 2 before evaluating the failing ratio, so a single misclassified SP is enough to cross the 50% threshold.
2. Per-SP is_failing classification comes purely from a rolling 1h delta of a maxMerge(value_max) counter, not raw event outcomes: a provider counts as failing when t >= 5 AND s/t < 0.7 in that window.
3. Cross-checking raw dealbot-worker pod logs for the flagged provider showed genuine successful retrievals landing during the alert window that did not increment the counter for ~100 minutes, while one small/fast failure did land immediately — so a provider that was actually succeeding read as 100% failing.
4. Fleet-wide jobs_completed_total for retrieval_sampled climbed continuously through the window, confirming the underlying check pipeline was healthy the whole time. The alert was purely a metric-export/aggregation-lag artifact.

This chart's query has no floor on absolute event volume or on failing_sps count — only active_sps >= 2 — so the same mechanism (tiny approved-SP fleet + counter lag) can plausibly trip any of the other 3 alerts too, not just sampledPieceRetrievalStatus.

[silent-cipher]

Did some more digging into this:

Summary

The real reason behind Approved SPs fraction failling by check type alerts being noisy/flaky is we only have 2-3 approved SPs per check type, and the alert math doesn't hold up with a pool that small.

What to do about it

1. We can increase the window (current window = 1hr) so it's less sensitive to which SPs happen to be active in a given hour
2. Alert on each SP's own success rate directly, instead of "% of a tiny fleet failing."

[BigLep]

@silent-cipher : I think we can do some alarm work here. Rather than increase the window, what if we require this check to require >= 66% of SPs to be failing. In a world with 3 SPs, that means all 3 are failing. (You could also argue for checking if we have 100% failure, but I went with 66% so that when we get more SPs we would trigger with less. Ideally when we have more SPs we'd adjust the threshold anyways, but 66% feels a little more future proof than 100%).

If some reason taht doesn't work, then lets increase the window.

I definitely want to take care of the noisy alarm. If we see it too uch we'll start to ignore.

[silent-cipher]

@BigLep, thanks for pushing this forward. I dug into the 66 % threshold using the actual data and found a problem, so I wanted to flag it before we implement anything.

• With three SPs, two failures give 2/3 = 66.67 %, which is still ≥ 66 %. A literal 66 % threshold therefore triggers when two of three SPs fail, not when all three do. To require unanimity you’d need a threshold of about 67 %–75 % for 3 SP pool.
• More importantly, any threshold that requires two or more SPs to fail would miss the actual outage that has been driving this alert. The alert has mostly fired because a single SP has been failing data‑storage status checks continuously for the past couple of weeks, while the one or two approved SPs on that check remained healthy.
• I recommend using your fallback: increase the window and keep the threshold at 50 %. This adjustment is safe; it doesn’t affect the “how many SPs” logic, so it won’t suppress detection of genuine single‑SP failures.