feat(mongodb): alert when compaction is needed

[delthas]

Summary

Follow-up to ZENKO-5285 (PR #2431), which bundled two alerts in its description and only shipped the first (createIndexes-failed). This adds the second: a fragmentation/compaction-needed signal.

Two commits, deliberately split:

1. mongodb: enable dbstatsfreestorage collector in exporter — one-line values.yaml change adding --collector.dbstatsfreestorage to metrics.extraArgs. The chart's exporter no longer uses --collect-all (dropped in 8414833 for ZENKO-5281); each sub-collector is opted into individually in extraArgs, which already lists dbstats, diagnosticdata, replicasetstatus, and topmetrics. We need dbstatsfreestorage on that list so the freeStorageSize / indexFreeStorageSize / totalFreeStorageSize fields are emitted as clean top-level mongodb_dbstats_* series instead of being buried in the per-host mongodb_dbstats_raw_<host>_* expansion.
2. mongodb: alert when compaction is needed — the alert proper.

The alert

(mongodb_dbstats_fsUsedSize / mongodb_dbstats_fsTotalSize) > 0.7
AND (mongodb_dbstats_totalFreeStorageSize / mongodb_dbstats_totalSize) > 0.3
AND mongodb_dbstats_totalFreeStorageSize > 10 GiB
for: 1h

Per-(pod, database) granularity. Severity warning.

Heuristic — open for discussion

All three conditions and the for: are exposed as x-inputs (compactionDiskUsageThreshold, compactionFreeStorageRatioThreshold, compactionFreeStorageAbsoluteThreshold). The defaults are my best guesses and very much up for review:

• Disk usage > 70%: only alert when compaction would actually relieve pressure. A pure-ratio alert without this gate is noisy: a freshly-loaded test cluster currently has a bucket DB at 28.6% free-storage ratio on a 13%-full disk — nothing to do there. Still a value that makes sense?
• Free ratio > 30%: standard rule-of-thumb for "reclaim is meaningful". 50% would be more conservative.
• Absolute > 10 GiB: floor to avoid alerting on tiny DBs where even 90% reclaimable is only a few MB. 5 GiB would also be reasonable.
• for: 1h: fragmentation builds slowly. Anything shorter would alert on transient writes.

Per-DB granularity means the alert tells you pod={{ "{{ $labels.pod }}" }} and database={{ "{{ $labels.database }}" }} but does not identify the specific collection. To find that, an operator runs collStats on the alerting DB. The exporter has a --collector.collstats flag we could enable to get per-collection visibility, but on Artesca clusters with thousands of buckets per DB that's a real cardinality cost — deferred for now.

Safety against missing / zero values

• Missing metric for a (pod, database) tuple → PromQL produces empty for that tuple → no alert. ✓
• totalFreeStorageSize = 0 (no fragmentation) → ratio is 0, fails both ratio and absolute checks. ✓
• totalSize = 0 (empty DB) → 0/0 = NaN, NaN comparisons are always false. And totalFreeStorageSize can't exceed totalSize, so the absolute check fails too. ✓
• The absolute-bytes condition acts as defense in depth against any divide-by-zero corner case producing +Inf on the ratio leg.

Why this was split off from ZENKO-5285

The original PR shipped the simpler index-failure alert. This one needed more thought on the right signal, the per-(pod, database) aggregation, and the exporter flag dependency. Reviewer @DarkIsDude explicitly asked for a follow-up.

Related

• ZENKO-5293 — this ticket
• ZENKO-5285 — parent alert ticket; createIndexes-failed half shipped in feat(mongodb): alert on createIndexes command failures #2431
• BB-784 — Backbeat-side LifecycleIndexCreationFailed (PR feat(lifecycle): alert when index creation fails backbeat#2753)
• BB-783 — putBucketIndexesFailed metric (PR feat(lifecycle): emit putBucketIndexesFailed metric on index creation errors backbeat#2751)
• Parent epic ARTESCA-16740 — Empty bucket feature

Issue: ZENKO-5293

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[delthas]

@francoisferrand Heuristics are up for discussion -- I chose a mix of:

• Disk is being significantly used (to avoid saving 10GB on a disk that has 10TB free space)
• There is a signficiant number of GBs to save (to avoid alarms on tiny collections)
• There is a significant ratio of data to save (to avoid running compact just to save 10GB on a 10TB DB)

[francoisferrand]

• not sure about thresholds/computation: are these relevant default?
• not sure we should merge in 2.15, or target 2.16 and get time to "preview" this and avoid shipping alerts which would lead to support calls...

[francoisferrand]

are we confident about these levels ?

• good defaults everywhere
• alert when really needed
• not too noisy

these alerts need to be "low touch", and fire only when an action is required...

[francoisferrand]

(note: since we add config, shoud add matching options in ZKOP afterwards, to allow setting them...)

[francoisferrand]

not sure about this rule : percentage of disk (FS) and percentage of reclaimable (I gather totalFreeStorageSize is what we can reclaim?) could fire in many case, but often when not relevant ?

e.g. we are interested when the reclaimable amount is significant vs the available mongodb (free/total) size I guess?

• i.e. your last condition, or possibly a variable where the threshold is a % of fs(free/total)size?
• first condition seems fine, to avoid alerts when there is lot of space (though that is debatable, not sure it is really useful or we want to delay in that case)
• I don't see at all what the benefit of % of reclaimable space is ?

[bert-e]

Conflict

There is a conflict between your branch improvement/ZENKO-5293/mongodb-compaction-needed-alert and the
destination branch development/2.15.

Please resolve the conflict on the feature branch (improvement/ZENKO-5293/mongodb-compaction-needed-alert).

git fetch && \
git checkout origin/improvement/ZENKO-5293/mongodb-compaction-needed-alert && \
git merge origin/development/2.15

Resolve merge conflicts and commit

git push origin HEAD:improvement/ZENKO-5293/mongodb-compaction-needed-alert