[emgarten]

Functional test runs that stop before they finish leave sleet-test-{guid} buckets in the AWS test account, because the process that would delete them is gone. Cancelled CI runs are rare in the workflow history, so most of the leftovers probably come from:

• stopped local or IDE runs
• runs killed by the 20-minute hang dump
• cleanups that skipped a bucket because AWS briefly reported a new bucket as not found

Changes

• A new test, GivenAS3AccountVerifyStaleTestBucketsAreDeleted, deletes the sleet-test-{guid} buckets in the account that are more than 2 hours old.
  • 2 hours is longer than the 60-minute cloud job timeout, so the buckets of runs in progress are kept.
  • It runs on every S3 target, so each run deletes what earlier runs left behind.
• Test bucket cleanup now:
  • deletes objects in batches of 1,000, the most that one DeleteObjects request allows
  • retries failures, such as a conflict on a bucket that was just created
  • checks a bucket that's not found a second time before it treats it as deleted
  • tries again from DisposeAsync if an earlier CleanupAsync failed, and disposes the local cache even if cleanup throws
• CONTRIBUTING.md says that stopped runs can leave buckets behind, and that the S3 test credentials need s3:ListAllMyBuckets.

While a bucket is being created, RustFS lists it with a creation date of 1970-01-01. The sweep treats that date as unknown, because otherwise it deletes the buckets of tests that are running.

Before merging

The AWS test credentials need s3:ListAllMyBuckets. Without it, the sweep test fails with AccessDenied.

Validation

• build.ps1 passes.
• functional-tests.ps1 -Target rustfs passed 3 times, 24 of 24 tests each time, with no buckets left in RustFS.
• With the stale age temporarily set to 5 seconds, the sweep deleted 3 test buckets on RustFS, one with 2,500 objects. It kept 2 buckets whose names don't match, sleet-test-feed and keep-{guid}.
• AWS and R2 weren't tested, since there are no credentials for them locally. The R2 docs list support for every S3 call the cleanup and sweep use.

Azure containers aren't swept yet.