Skip to content

feat: backport bulk skip-WAL support and integration coverage - #9627

Merged
v0y4g3r merged 2 commits into
GreptimeTeam:release/5e02efa9from
v0y4g3r:backport/request-skip-wal-bulk
Oct 16, 2026
Merged

v0y4g3r merged 2 commits into
GreptimeTeam:release/5e02efa9from
v0y4g3r:backport/request-skip-wal-bulk

Conversation

@v0y4g3r

@v0y4g3r v0y4g3r commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Restore the missing skip-WAL prerequisites rather than maintaining a reduced close-flush backport around their absence.

The first commit copies #9093's integration harness and tests without functional changes. The second ports #9110: forwards the insert_skip_wal hint through Flight DoPut and bulk region RPCs, keeps different WAL policies in separate pending Prometheus batches, and skips bulk WAL entry construction while retaining memtable writes, sequence assignment, and completion notifications. Metric metadata writes remain WAL-backed. The proto dependency advances to the original #9110 revision 0c4beffaa2b3968034915867593a717c6ab0c6b4 (greptime-proto#339).

Release-specific adaptations

The release branch already has the newer shared write dispatcher and partition-column preservation backports. Preserve them instead of restoring main's older direct-dispatch code:

  • Carry skip_wal on both initial bulk dispatch paths and through request rebuilding after partition-route changes.
  • Add true/false policy assertions to the existing bulk repartition/unchanged-route regression.
  • Keep partition versions and physical partition columns in pending batcher requests, alongside the new policy flag.
  • Retain the physical bulk partition-column regression and exercise both WAL policies.
  • Initialize the new field in release-only staging/dispatcher test fixtures.

No persisted WAL/schema encoding changes. The additive protobuf flag defaults to false when omitted. Skip-WAL data remains vulnerable to loss before flush; this PR does not include the close-flush change from #9440.

Validation

  • test: cover request-level insert WAL skipping end to end #9093 alone: focused skip-WAL protocol/COPY integration selection, 27 passed.
  • Combined affected packages (client, frontend, operator, servers, store-api, mito2, metric-engine) plus skip-WAL/Flight integration selection: 2491 passed, 810 filtered/ignored tests not run.
  • cargo sqlness bare -t '(session_skip_wal|copy_skip_wal)': 3 cases passed (session in standalone/distributed, COPY in standalone). Result files remain identical to the upstream-generated snapshots.
  • Affected-package Clippy with --all-targets --no-deps -- -D warnings: passed.
  • Formatting and locked metadata passed. Source diff checks pass; the two unchanged upstream-generated .result snapshots retain their terminal blank lines.

Full workspace tests, cross-version compatibility runs, and external-service Kafka runs were not executed. The distributed integration harness uses in-process services, not process-crash experiments.

PR Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.
  • This PR needs to be backported to release branches, and I have added the backport-<target> labels (e.g. backport-v1.3 targets release/v1.3).

Backport GreptimeTeam#9093 as the shared protocol and storage test harness prerequisite for bulk request WAL skipping.

Signed-off-by: WenyXu <wenymedia@gmail.com>

(cherry picked from commit a673e08)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Backport GreptimeTeam#9110 with its protocol update and Flight/Prometheus coverage. Keep the release branch's write dispatcher and partition-column retention, including the skip-WAL flag when retry routing rebuilds bulk requests.

Signed-off-by: WenyXu <wenymedia@gmail.com>

(cherry picked from commit 7370257)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 16:51
@v0y4g3r
v0y4g3r requested review from a team, WenyXu and evenyag as code owners October 15, 2026 16:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

Copy link
Copy Markdown
Contributor

Warning

@v0y4g3r has 6 open non-draft pull requests in this repository, over the limit of 5.

Review is the scarcest resource here. Please land or close some of these before
pushing this one forward.

View your open pull requests

This check is advisory for now and blocks nothing.

@github-actions github-actions Bot added the docs-required This change requires docs update. label Oct 15, 2026
@v0y4g3r
v0y4g3r merged commit 12b7ebe into GreptimeTeam:release/5e02efa9 Oct 16, 2026
42 checks passed
v0y4g3r added a commit to v0y4g3r/greptimedb that referenced this pull request Oct 16, 2026
Merge release/5e02efa9 after GreptimeTeam#9627 and GreptimeTeam#9624. Resolve the worker test conflict using the shared memtable installation path, honor request-level bulk skip-WAL flags, and restore bulk close/reopen and WAL-only recovery coverage while preserving release-specific partition-column tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-required This change requires docs update. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants