Skip to content

test: harden flaky tests that are no longer retried on CI - #922

Merged
turnipdabeets merged 1 commit into
mainfrom
test/harden-flaky-tests
Oct 8, 2026
Merged

turnipdabeets merged 1 commit into
mainfrom
test/harden-flaky-tests

Conversation

@turnipdabeets

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

CI retries failed XCTest cases on the iOS simulator job (#882), but Swift Testing failures aren't retried. Since #915 moved the remaining specs to Swift Testing, these tests now fail CI outright when they flake:

  • ExceptionSteps: storage.reset() keeps the event queue, so a previous test's $exception could still be on disk, get sent, and be read as events.first. getSut now deletes this token's app folder first. It's scoped to the token because some tests run two SDKs at once.
  • QueueTest:
    • A request that never arrives, or an extra upload, now records an issue instead of indexing out of range. Before, one late request crashed the whole test process.
    • The fixture sets server.batchProjectToken so it only counts its own queue's /batch requests. The stub is process-global, so a stray request from another SDK instance shifted the request count.
  • ApplicationViewLayoutPublisher: waitUntil timeout raised from 1s to 5s. It returns as soon as the condition holds, so the higher ceiling only matters on a slow runner.

The replay queue migration test flake is fixed separately in #920.

💚 How did you test it?

  • make test filtered to PostHogExceptionStepsTest, PostHogQueueTest and PostHogQueueUploadDispositionTest: 35 tests pass on macOS.
  • ApplicationViewLayoutPublisherTest is iOS-only; the change only raises the timeout default. Covered by the test-ios-simulator job.
  • SwiftFormat and SwiftLint clean on the changed files.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written with Claude Code. Test-only changes, so there's no changeset.

🤖 Generated with Claude Code

Swift Testing failures aren't retried on the iOS simulator job, so these now
fail CI outright:

- ExceptionSteps: clear this token's folder in getSut; a previous test's
  $exception could be re-sent and read as events.first.
- QueueTest: record an issue instead of trapping on a missing request or
  extra upload, and only count this queue's /batch requests.
- ApplicationViewLayoutPublisher: raise the wait ceiling from 1s to 5s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@turnipdabeets
turnipdabeets requested a review from a team as a code owner October 7, 2026 20:59
@turnipdabeets turnipdabeets self-assigned this Oct 7, 2026
@turnipdabeets
turnipdabeets merged commit f737c73 into main Oct 8, 2026
47 checks passed
@turnipdabeets
turnipdabeets deleted the test/harden-flaky-tests branch October 8, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants