feat(gguf): the GGUF v1 toy — the smallest population, and the cleanest

[ciresnave-bot]

tinyllamas-stories-260k-f32.gguf (512 tokens) is the fourth and last SentencePiece vocabulary in the corpus. It has no oracle: no corpus file shares or extends its vocabulary, so its warrant is the same shape as Baichuan's and rests on ~334x fewer observations. That is the honest caveat, and it is in the warrant string a user reads.

Measured

v1 toy   512 tokens   164 adjacent pairs   150 INFORMATIVE   14 ties (8.5%)
         0 raw / 0 genuine order violations
         forced (id 259): genuine 1

Taken at 69b42d8 (branched from main at c15ec32, which carries #81), from the SPM suite: 16 passed; 0 failed; 718 filtered out; finished in 13.88s.

SMALL IS NOT WEAK, and the distinction is measured rather than argued

At 8.5% ties this file has the highest informative fraction of the four allowlisted vocabularies:

vocabulary	adjacent pairs	ties
v1 toy	164	8.5%
baichuan	54803	44.4%
llama-spm	61248	51.7%

So its limitation is population size alone, not population quality. Those are separately actionable — size is fixed by more data, quality by a better measurement — and conflating them would discount the wrong thing.

The "small but clean" argument is itself guarded: the test asserts the tie fraction stays under 20%. If this file ever loses that property the warrant becomes false and has to be rewritten rather than kept.

The negative case is in the claim

On 164 pairs it is the whole point: a broken comparator returns 0 here too. The forcing arm is the only thing that tells those apart, so it is asserted rather than mentioned — forced (id 259): genuine 1, and the assertion is forced_genuine > v.genuine, not a bare nonzero. The warrant string carries PROOF AGAINST GROSS FAILURE, NOT PROOF OF CORRECTNESS.

Corpus sweep

30 files: 22 rebuilt, 8 refused  ->  23 rebuilt, 7 refused

DELETED: a_vocabulary_with_no_oracle_is_still_refused

It walked corpus files that were not allowlisted and checked each was refused with its digest named. Every SentencePiece vocabulary in the corpus is now allowlisted, so it had no subjects left — and a for over an empty list reaches no assertion and reports success. A guard that outlives its subject is a green light with nothing behind it. Its own comment said to delete it rather than rewrite it over an empty set.

⚠️ What that costs, stated rather than quietly lost: the refusal message path is no longer reachable from any corpus file, because reaching it needs a llama-model checkpoint with no merges whose digest is absent from the allowlist, and the corpus no longer contains one.

Replaced with the_derivation_allowlist_is_closed_and_its_lookup_discriminates, which needs no corpus and so runs in the ordinary suite. It asserts all four known digests are allowlisted — the control, without which the negative case would pass against a function that returns None for everything — and that four absent digests are not, including one differing from a real entry by a single character.

Verification status

Nine local gate steps completed (fmt, lockfile, doc, test, featgate, clippy, spm, sweep, e2e), exit 0; lib suite 712 passed; 0 failed. ⚠️ That chain is my own instrument and its scope is this machine — it is not the forge's verdict. CI and Codacy decide whether this PR is green, and I am not calling it green from a local run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JtTf3315ahKPPyBiYugnPh

Summary by Sourcery

Allow the GGUF SentencePiece pipeline to recognize and validate the small, clean v1 toy vocabulary while replacing an obsolete corpus refusal guard with discriminating allowlist coverage.

New Features:

• Add support and an explicit derivation warrant for the 512-token GGUF v1 toy vocabulary.
• Add corpus-backed tests covering the toy vocabulary’s rebuild, score-order agreement, population quality, and comparator failure detection.

Bug Fixes:

• Replace the vacuous refusal guard with meaningful allowlist coverage and negative lookup checks.

Enhancements:

• Document the toy vocabulary’s smaller but more informative evidence population and its weaker, non-oracle warrant.
• Strengthen SentencePiece allowlist tests by verifying all known entries resolve and near-miss or absent digests are rejected.

Tests:

• Add assertions for the toy vocabulary’s tie-rate threshold, forced genuine violation, and tokenizer usability.
• Remove the obsolete test for refusing an unallowlisted corpus vocabulary once all corpus vocabularies became allowlisted.

[sourcery-ai]

Sorry @ciresnave-bot, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 20 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

The PR allowlists and validates the smallest GGUF v1 vocabulary, including an explicit small-but-clean evidence warrant, comparator forcing check, rebuild smoke test, and corpus-independent allowlist discrimination. It also replaces a now-vacuous refusal guard with closed-table tests while documenting that unknown-vocabulary refusal messaging is no longer covered by corpus fixtures.

Sequence diagram for validating the GGUF v1 toy vocabulary

sequenceDiagram
    participant Test as V1 toy tests
    participant Corpus as GGUF corpus
    participant Content
    participant Comparator as score_order_violations
    participant Tokenizer as extract_tokenizer

    Test->>Corpus: read_corpus("tinyllamas-stories-260k-f32.gguf")
    Corpus-->>Test: GGUF content
    Test->>Content: vocab_sha256(tokens)
    Content-->>Test: V1_TOY_VOCAB_SHA256
    Test->>Comparator: score_order_violations(tokens, scores)
    Comparator-->>Test: 0 genuine violations
    Test->>Comparator: score_order_violations(tokens, forced scores)
    Comparator-->>Test: forced_genuine > 0
    Test->>Tokenizer: extract_tokenizer()
    Tokenizer-->>Test: rebuilt tokenizer
    Test->>Tokenizer: encode(probe, false)
    Tokenizer-->>Test: non-empty token IDs

Loading

Flow diagram for GGUF derivation allowlist discrimination

flowchart TD
    A[Known vocabulary digests] --> B[Content.sp m_derivation_warrant]
    B --> C{Digest is allowlisted?}
    C -->|Yes| D[Return derivation warrant]
    C -->|No| E[Return None]
    F[Absent digests] --> B
    G[Single-character near-miss digest] --> B
    D --> H[Closed allowlist control passes]
    E --> I[Unknown vocabulary remains unverified]

Loading

File-Level Changes

Change	Details	Files
Adds the 512-token GGUF v1 vocabulary to the SentencePiece derivation allowlist with an explicit, caveated warrant.	Registers the vocabulary SHA-256 digest and documents its no-oracle, small-population evidence limits. Adds a warrant describing agreement with converter scores, tie population, forcing validation, and the distinction between proof against gross failure and correctness. Preserves v1 compatibility through candle while noting the project parser rejects v1.	src/gguf/mod.rs
Adds corpus-independent allowlist coverage and corpus-gated validation for the new vocabulary.	Replaces the vacuous refusal-loop test with positive checks for all four allowlisted digests and negative checks for absent and near-miss digests. Verifies the v1 toy's derived merge order against scores, enforces a sub-20% tie fraction, and requires a deliberately forced violation to be detected. Verifies the production tokenizer rebuild path and basic encoding probes for the v1 file.	src/gguf/mod.rs
Removes the obsolete unallowlisted-corpus refusal test and documents the resulting coverage boundary.	Deletes the test whose empty subject set could pass without assertions. Records that the refusal-message path is no longer exercised by the available corpus.	src/gguf/mod.rs

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}