Detect reorgs that occur while the server is down#563
Open
frankbraun wants to merge 1 commit into
Open
Conversation
NewBlockCache did not initialize latestHash from the last block in the disk cache, so HashMatch vacuously accepted the first block ingested after every restart. If the backing node reorged across a lightwalletd restart, the new chain was appended on top of the orphaned blocks, which then remained in the compact-block cache permanently, serving wallets an internally inconsistent block stream. Initialize latestHash in NewBlockCache (using the existing setLatestHash helper, previously only called from Reorg) so the ingestor walks back orphaned blocks on the first ingest after a restart, the same way it does for reorgs detected while running. This also avoids the unnecessary drop and re-fetch of the cache tip on every restart when no reorg occurred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ValarDragon
added a commit
to valargroup/lightwalletd
that referenced
this pull request
Jun 11, 2026
…eorg Backport upstream zcash#563: Detect reorgs that occur while the server is down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NewBlockCacherebuilds the block index from the on-disk cache but never initializeslatestHash(thesetLatestHashhelper exists but is only called fromReorg). SinceHashMatchvacuously accepts any block whilelatestHash == hash32.Nil, the first block ingested after every restart is appended without a prev-hash check against the cache tip.Consequence: if the backing node reorgs across a lightwalletd restart, the new chain is silently appended on top of the orphaned blocks. The orphans remain in the append-only compact-block cache permanently, and the server hands out an internally inconsistent block stream (block N's
prevHashdoesn't match the block N−1 it actually serves).Real-world impact
This happened in production during the 2026-06-03 mainnet reorg: the zebrad instance backing zec.fyi followed a minority branch for heights 3364600–3364603 and was restarted while stuck on it; it came back on the canonical chain, and lightwalletd (restarted at the same time) appended canonical 3364604 directly on top of the orphaned 3364603 — no
REORG:log line, no unwind. The server then served the four orphaned blocks, breaking prev-hash continuity at 3364604, until the cache was manually repaired a week later. Wallets that validate chain continuity stall on such a stream; wallets that don't can scan orphaned blocks.Fix
Call the existing
setLatestHash()at the end ofNewBlockCache. WithlatestHashrestored from disk, the first ingest after a restart goes through the sameHashMatch→Reorgwalk-back as reorgs detected while running. A side benefit: when no reorg occurred, startup no longer drops and re-fetches the cache tip (previously,getbestblockhashcould never matchNil, forcing an unnecessaryREORG: dropping block …of the tip on every restart).Testing
TestCache: after reopening the cache from disk,latestHashmust equal the top block's hash andHashMatchmust reject a non-connecting block. The test fails on master (latestHash not initialized after restart) and passes with this change.go test ./...passes.🤖 Generated with Claude Code