Dealbot accumulates many active data sets per provider (calib: 196 on provider 23) — no reuse/cleanup

[SgtPooki]

Problem

On calibration, dealbot's wallet (0x305025…dca0b) holds a large and growing number of active (un-terminated, pdpEndEpoch=0n) data sets per provider, instead of reusing or cleaning them up:

provider	active dealbot data sets
23 (🚀, 0x6c8E…d796)	196
4, 6	25
2, 9	24
5, 8, 15, 22, 27	15
26	7
24 (sp-playground)	0 (terminates all → contrast)

Provider 23's 196: 182 share identical metadata {"source":"dealbot","withIPFSIndexing":""} (data set ids 13115–13751), 14 are unique dealbotDS:N indexed slots; all have pieces, zero terminated. (Source: Foc-observer fwss_data_set_created LEFT JOIN fwss_service_terminated.)

Why it matters

StorageContext.resolveByProviderId (synapse-sdk) scans every active data set for the provider on each deal to metadata-select one. So this count is the per-resolve RPC fan-out width — provider 23 alone forces ~132 eth_calls per resolve and it grows daily. See the RPC-burst tracking issue. (Also wastes storage/payment rails.)

Open question (to resolve)

Are the 182 identical-metadata sets a reuse failure (resolveByProviderId should return the existing metadata-matching set, but dealbot creates a new one anyway) or intended fresh-create by data_set_creation without cleanup? Provider 24 terminates everything; provider 23 terminates nothing — so cleanup is at least inconsistent.

Next steps

• Determine the create-vs-reuse path for the base-metadata sets (instrument createContext/createDataSetWithPiece).
• Decide policy: reuse the existing match, or terminate old test data sets (and why provider 23 never terminates while 24 does).
• Sweep the existing accumulated sets on calib.

Done criteria

• Active dealbot data-set count per provider stays bounded on calib.
• Root of the 182-set accumulation identified and fixed (reuse or cleanup).

[beck-8]

23 is my problem. Node 23 uses glif's RPC, and now it encounters the too many pending messages for actor issue (unable to send too many messages). Waiting for Lotus to release the release, and the glif to upgrade. after I don't think I have this problem.

[SgtPooki]

thanks for the info @beck-8

I think there is also work here to ensure that dealbot is not leaving a bunch of data-sets in created state. dealbot should be able to see this and terminate some data-sets if they're accumulating (I need to get #548 over the line)

[BigLep]

Triage comment from 2026 FOC Board Triage (Post GA): we should fix the fact that dealbot can accumulate data-sets without them cleaning up, but the root cause of this particular issue is an SP bug. However, this can cause dealbot spend unnecessarily.. there is a draft PR out to address this

[SgtPooki]

Root cause found. The 182 base-metadata sets are debris from a single 43-hour incident, not an ongoing reuse bug.

Mechanism: synapse-core 0.4.1 rewrote getClientDataSets to paginate with a default limit of 100, and the accumulator loop treats limit as a cap on total results, not page size. resolveByProviderId doesn't pass a limit, so it silently saw only the wallet's oldest 100 data sets. The wallet had 150 sets at the time; providers 23 and 24 were the newest (first sets at positions 121 and 136), so their sets were invisible to resolution. Every deal tick created a new set, which appended past the cap and stayed invisible.

Evidence:

• 181/182 sets created 2026-05-05 12:00 to 2026-05-07 07:00 UTC at 4-5/hour (the deal-check rate), then dead stop. Provider 24 burst identically (182 sets, interleaved ids) but self-terminates, hence its 0 active count. Mainnet provider 13 got 144, all since terminated.
• Window bracketed exactly by chore: upgrade synapse to latest #497 (upgrade to sdk 0.40.4 / core 0.4.1) and fix: revert back to old synapse version #512 (revert, merged 07:27 UTC May 7, the hour the burst stops).
• resolveByProviderId itself is byte-identical in 0.40.2 and 1.0.1; core 0.7.0 (current) defaults to fetch-all, so steady-state reuse works today. Zero accumulation since May 7 confirms it.

Revised next steps:

1. One-time sweep: terminate the still-active burst sets on calib. Current list: provider 23: 181, providers 4/6: 10 each, providers 2/9: 9 each (219 total, all created inside the window). Also shrinks the Calibration eRPC 429 storms: deal flow resolveByProviderId fans out O(active data sets) eth_calls per deal #606 resolve fan-out.
2. Alert on data-set creation rate per provider. This ran for 43 hours; an alert would have caught it in one.

No dealbot code change needed for create-vs-reuse.

[BigLep]

@beck-8 : can you please take this?