warp sync to finney mainnet never leaves block 0: "invalid commit in grandpa justification"

[JSONbored]

Summary

--sync=warp against finney mainnet never leaves block 0. The node requests a warp proof beginning at genesis, peers serve it, and verification fails with:

Bad warp proof response: bad justification for header: invalid commit in grandpa justification

The node then applies a -536870912 reputation penalty, drops the peer, and repeats against the next one indefinitely. Status stays ⏩ Warping, Downloading finality proofs, 0.00 Mib forever, at any peer count.

Reproduced independently on two unrelated hosts in different datacentres and on different providers.

Reproduce

--base-path=/data
--chain=/home/subtensor/chainspecs/raw_spec_finney.json
--sync=warp
--rpc-external
--rpc-cors=all

Reproduces with a completely minimal flag set (above) and with a tuned one; not configuration-dependent.

Trace

With --log=sync=trace:

TRACE sync: New WarpProofRequest to 12D3KooWErnA…, begin hash: 0x2f05…6c03.
TRACE sync: Created `WarpProofRequest` to 12D3KooWErnA…,
      request: WarpProofRequest { begin: 0x2f0555cc76fc… }
DEBUG sync: Bad warp proof response: bad justification for header:
      invalid commit in grandpa justification
TRACE sync: PeerId("12D3KooWErnA…") dropped:
      ReputationChange { value: -536870912, reason: "Bad warp proof" }

begin is the finney genesis hash 0x2f0555cc76fc2840a25a6ea3b9637146806f1f44b090c175ffde2a7e5ab36c03.

A large fraction of peers additionally answer Request to peer … failed: Refused. — expected, since a warp-synced node has no justification history to serve from — but the peers that do serve a proof all fail verification identically.

Rejection counts by client version

Same host, same flags, DB wiped between runs:

image	built	bad warp proofs	distinct peers penalised
ghcr.io/raofoundation/subtensor:latest (= v432)	2026-07-17	151	14
ghcr.io/opentensor/subtensor:latest	2026-06-29	40	13
ghcr.io/raofoundation/subtensor:main	2026-08-21	still fails, same error	—

Suspected cause

node/src/service/grandpa_warp_sync.rs gives mainnet and testnet materially different treatment:

if genesis_hash == TESTNET_GENESIS {
    Config::TestnetCheckpoints(testnet_checkpoints())
} else if genesis_hash == FINNEY_GENESIS {
    Config::OneTimeInitialSetId(3)
}

Testnet gets a full Vec<AuthoritySetHardFork<Block>> checkpoint list — added recently in #3055, #3082, 08762343 fix(testnet): use historical GRANDPA signing sets, add2b31a fix(testnet): correct GRANDPA checkpoint transition.

Finney gets a single initial set-ID offset, applied only when set_id == 0, with no hard-fork checkpoints at all. If finney has any authority-set discontinuity past genesis, that path has nothing to reconcile it — which matches invalid commit in grandpa justification on a proof that begins at genesis.

The testnet fixes appear to address this exact class of failure; mainnet looks like it has not received the equivalent treatment. There is also no CLI escape hatch — the config is selected by hardcoded genesis hash, so an operator cannot supply finney checkpoints.

Ruled out

• Wrong network. Our genesis is 0x2f0555cc76fc2840a25a6ea3b9637146806f1f44b090c175ffde2a7e5ab36c03, byte-identical to chain_getBlockHash(0) from entrypoint-finney.opentensor.ai. system_chain = Bittensor. Absolute chainspec path used, never --chain=finney.
• Stalled finality. Finney is finalizing normally — finalized 8900556 vs best 8900558 at time of test.
• Peering. Peers report bestNumber=8900412 (chain head). Reproduced at 40 peers.
• Egress/firewall. Container-to-bootnode TCP verified from inside the Docker bridge.
• Database backend. Identical behaviour with --database=paritydb and with RocksDB.
• Our tuning. Reproduces on a minimal flag set.
• --sync=fast. Advertised in --help but rejected at runtime with Service(Other("Unsupported sync mode")), so warp is the only practical option — full is a multi-week genesis sync.

Separate observation: litep2p

Also seen while narrowing this down, possibly worth its own issue:

• On one host, --network-backend=litep2p (the default on :main) connected 0 peers indefinitely, while --network-backend=libp2p on the same host, same image, same config reached 30 peers in ~150s.
• On the other host litep2p peered fine but produced 55 bad warp proofs where libp2p produced 0–3.

Impact

Warp is the documented way to stand up a lite node, and it is the only fast path — fast is unimplemented and full is a multi-week genesis sync. With warp broken on mainnet, there is no practical route to a new finney lite node, and no public snapshots exist.

Happy to supply full trace logs or test a patch.

[mkratt798-cmyk]

I have faced same issue

[JSONbored]

Root cause confirmed, with the complete missing data.

We binary-searched GrandpaApi_current_set_id over an archive endpoint and mapped every GRANDPA authority-set transition in finney's history:

set	first block	hash
1	5,615,167	0x78b6189f8e0fa11f71c793281104f1f65112b11b508b30d12760e4f42391c6f2
3	5,642,728	0xb67e1941cf3f4f4df0ce45d4c03a311866e2a44a6170673dca812adc16a984e3
4	8,867,015	0xb59a08768fbf83766fa1609329058f72b5bb47ab6c4e155ef64561fbe0632e26
5	8,867,349	0xc5e94627d67544078e65597a06fc70bd6864c1cb09f351c30116c162441790e9
6	8,867,448	0x511948e96e1d479d0a92d89bb976638780f2c65a93a5d5be710f22ee15c60200

Two things stand out:

1. Set 2 is skipped (1 → 3 at block 5,642,728) — presumably the anomaly OneTimeInitialSetId(3) was papering over.
2. Sets 4, 5 and 6 all occurred within 433 blocks on 2026-08-18 — which is exactly when fresh warp syncs stopped working network-wide. Any node synced before that date works; any node syncing after cannot, because OneTimeInitialSetId(3) knows nothing past set 3.

So finney needs the same treatment testnet received in #3082/#3055: a hard-fork checkpoint list. The grandpa_authorities at each transition hash above (all 20-member sets, weight 1) are retrievable via state_call GrandpaApi_grandpa_authorities at those hashes on any archive node — happy to paste the full lists if useful.

We are testing a local build carrying exactly these checkpoints as AuthoritySetHardFork entries and will report whether warp completes against mainnet.

Method note for reproducibility: state_call GrandpaApi_current_set_id "0x" <blockhash> on an archive endpoint, binary-searched from genesis to head — no special access required.

[JSONbored]

Follow-up: the checkpoints alone do not fix warp — and the same gap breaks --sync=fast too

Reporting back on the local build we said we would test, plus a finding that widens the scope of this issue.

1. Warp still fails with the checkpoints loaded

We built AuthoritySetHardFork entries for all five transitions from the table above (sets 1, 3, 4, 5, 6) and returned them through Config::TestnetCheckpoints(...) — the same path testnet uses. The node confirms them at startup:

Testnet GRANDPA warp sync checkpoints enabled.

Behaviour improves markedly — bad-proof count in the first minutes drops from 19–69 down to 1 — but warp still never progresses:

⏩ Warping, Downloading finality proofs, 0.00 Mib (40 peers), best: #0 (0x2f05…6c03)

4 minutes, 40 peers, 0.00 MiB, currentBlock: 0.

So the checkpoint list is either necessary-but-not-sufficient, or our reconstruction of it is subtly wrong. Two candidates we cannot distinguish from outside the implementation:

1. Authority weights / ordering. We decode each set from state_call GrandpaApi_grandpa_authorities at the transition hash and assign every authority weight 1. Upstream's testnet_checkpoints() instead reuses a single shared testnet_checkpoint_authorities() list across all entries, rather than a per-set list.
2. Block semantics. We use the first block observed at set N (binary search over GrandpaApi_current_set_id). If AuthoritySetHardFork.block is meant to be the block at which the change is enacted rather than first observed, then every entry we built is off by one.

Knowing which of those two is correct would probably settle it, and we are happy to re-test.

2. The bigger finding: this is not warp-only — --sync=fast and --sync=fast-unsafe are broken by the same gap

This does not appear to be a warp-specific bug. Fast sync downloads the state completely and successfully, and then fails at the import step:

22:06:56  ⚙️ State sync, Importing state, 0%, 310.12 Mib (40 peers), best: #8910402 (0x6be9…8e7b)
22:07:20  Block import error: Potential long-range attack: block not in finalized chain.
22:07:20  💔 Error importing block 0xba549b2eaefbdb50b52d44a8eacb38ac39e2b55829ae927d51614be1a1d53cd0:
          consensus error: Import failed: Potential long-range attack: block not in finalized chain.

The node then restarts state sync from 0% and repeats indefinitely. We observed several full cycles: the state downloads to completion (~310 MiB), the anchor block is rejected, everything is discarded, and it begins again with a new target.

Root cause looks identical: under OneTimeInitialSetId(3) the node cannot verify finality past the 2026-08-18 transitions (sets 4/5/6 within 433 blocks), so it treats the state's anchor block as not being in the finalized chain.

Reproduced on three independent hosts, different providers and datacentres.

Consequence

On finney today, for a node syncing from scratch:

mode	result
--sync=warp	never leaves block 0
--sync=fast	state downloads, import rejected, loops forever
--sync=fast-unsafe	same as fast
--sync=full	works — it derives the authority sets by executing every block

--sync=full is the only mode that completes, and we measured it at ~0.65 blocks/sec at current chain height, i.e. ~118 days from genesis. In practice that means there is currently no viable path for a new operator to bring up a finney node without copying a database from an already-synced machine.

That seems worth reflecting in the issue title/scope — as written this reads as a warp bug, but fast sync is affected identically, and that is what removes the last usable option.

Offer

Happy to provide the full 20-member authority lists for sets 1/3/4/5/6 (retrievable by anyone via state_call GrandpaApi_grandpa_authorities at the hashes in the table above), or to build and test a candidate patch against mainnet and report back — we have hosts reproducing this reliably.

[JSONbored]

Root cause: warp seeds from the genesis authority list, so no set id or checkpoint can ever fix it

Following up on our own previous comment, which was wrong about the remedy. The
checkpoint table in it is still correct; the conclusion drawn from it was not.
Here is the actual mechanism, with the measurement behind it.

The chain has moved past the list the node starts from

GrandpaApi_current_set_id / GrandpaApi_grandpa_authorities at
chain_getFinalizedHead, taken from four independent archives:

endpoint	set_id	authorities sha256[:16]
archive.chain.opentensor.ai	6	347c5c9d786735b4
lite.chain.opentensor.ai	6	347c5c9d786735b4
entrypoint-finney.opentensor.ai	6	347c5c9d786735b4
bittensor-finney.api.onfinality.io	6	347c5c9d786735b4

All four agree. That 20-key list is disjoint from the one a fresh node holds.

Why the node holds the wrong one

Warp sync seeds itself from WarpSyncProvider::current_authorities(). For finney
that resolves to grandpa_link.shared_authority_set(), which on an empty
database is the chain spec genesis authority list, paired with the hardcoded
Config::OneTimeInitialSetId(3) in node/src/service/grandpa_warp_sync.rs.

Since sets 4, 5 and 6 landed on 2026-08-18, a peer that itself warp-synced after
that date holds no authority-set changes at all. The only proof it can
generate is a single final fragment justified by the current set. The joining
node verifies that fragment against set 3's keys, so no precommit validates, the
commit misses threshold, and validate_commit returns:

Bad warp proof response: bad justification for header: invalid commit in grandpa justification

from every peer in turn, each of which is then banned. This is self-reinforcing:
every node that warps successfully today becomes another peer that cannot serve
history tomorrow.

Why AuthoritySetHardFork cannot repair it

This is the part we got wrong last time, and it is worth stating plainly for
anyone else attempting the same fix. In warp_proof.rs, checkpoints are looked
up by the exact (hash, number) of a fragment header:

let is_checkpoint = if let Some((set_id, list)) =
    hard_forks.get_hard_forked_authorities(&(hash, number))

The fragment we need to authenticate is the peer's best finalized block, which
advances every 12 seconds. No statically pinned checkpoint can ever match it.
That is exactly why loading all five transitions dropped our bad-proof count but
never produced progress — the checkpoints matched nothing that was actually sent.

For the same reason, HardForks::new_initial_set_id(n) alone is insufficient: it
adjusts the set id while leaving the authority list at genesis, and the
signatures are checked against the list.

The fix: pin the set id and the authority list at warp start

Both halves have to move together. Minimal change, in
node/src/service/grandpa_warp_sync.rs:

pub(super) struct InitialSetIdProvider<P> {
    inner: P,
    initial_set_id: SetId,
    initial_authorities: Option<AuthorityList>,   // added
}

fn current_authorities(&self) -> AuthorityList {
    // This is what SEEDS warp sync. The inner provider returns the chain spec
    // GENESIS list on a fresh database, which no longer matches any set a peer
    // can prove.
    self.initial_authorities
        .clone()
        .unwrap_or_else(|| self.inner.current_authorities())
}

fed from the runtime call above so it can be refreshed after a rotation without a
rebuild, rather than hardcoding another constant that goes stale at the next one.

The trust model is identical to a chain spec lightSyncState: whoever supplies
the pair defines the warp root of trust, which is why we cross-checked it against
four independent archives before using it.

Confirmed working on finney

Built and run against raofoundation/subtensor with only the change above, on a
wiped database, --sync=warp, no checkpoints set:

👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
Finney GRANDPA warp sync one-time initial set ID patch enabled. Set ID = 6
Warp sync starting authorities pinned from environment: 20 authorities
⏩ Warping, Downloading finality proofs, 0.00 Mib (3 peers), best: #0, finalized #0
Warp sync is complete, continuing with state sync.

Ten seconds, zero bad proofs. The same box on the same image with the
authority list left at genesis produced nothing but invalid commit in grandpa justification from every peer. The set id alone was never enough — set id 6 with
the genesis list fails exactly like set id 3 does.

Suggested upstream shape

OneTimeInitialSetId(3) is a constant that silently expires at every authority
rotation, and it takes --sync=warp — the default sync mode — down with it when
it does. Two options that would not:

1. Populate lightSyncState in the finney chain spec and refresh it on release,
   which is the mechanism Substrate already provides for this. sync_state_genSyncSpec
   is not currently exposed on any subtensor endpoint we can reach, so this would
   also mean wiring sc-sync-state-rpc.
2. Keep the constant but make the (set_id, authorities) pair overridable at
   runtime, so operators can recover from a rotation without waiting on a release.

Happy to open a PR for whichever shape you prefer.