Skip to content

A v2 baseline fingerprint is bound to --no-llm: the meta stage rewrites five hashed Finding fields #124

Open
Open
A v2 baseline fingerprint is bound to --no-llm: the meta stage rewrites five hashed Finding fields#124
Labels
bugSomething isn't workingready-for-agentAgent: ready for Claude to pick up and work on

Description

@rodrigorjsf
rodrigorjsf
opened on Aug 13, 2026

Finding

The v2 baseline fingerprint is bound to --no-llm. A baseline generated with the flag suppresses
nothing on a scan run without it, and the other way round — the same failure mode
--spec-checks advisory had while it encoded itself as confidence = 0.0, on a flag that is far
more used. README.md recommends scan . --no-llm --baseline … as the CI shape, and
docs/SUPPRESSION.md documents skillspector baseline <path> [-o FILE] [--no-llm] … with no
caveat.

Reproduced deterministically, no LLM involved — one Finding with a code-example context, run
through the two paths meta_analyzer.node selects between (meta_analyzer.py:671
_fallback_filtered for --no-llm, meta_analyzer.py:796 _passthrough_with_defaults):

raw            sha256:eb98468d4ac27bbece8204d78012a45e2d61c298ae90400a80baabaa5b72667a
--no-llm path  sha256:6fcd51a7dc7e1b9a5e0632379b83a50128160850c1676902d661a9769f05d169 confidence 0.45
llm-fail path  sha256:a44f1908f736628fedc5ce99dbf90298d80bddffbb512fee3d7c2a87120649c8 confidence 0.9
equal? False

Reason

suppression.finding_fingerprint hashes confidence, message, remediation, explanation and
tags (payload at suppression.py:150-166). The meta-analysis stage writes every one of those
names, and which branch runs is decided by the flag:

  • meta_analyzer.py:288-291 — confidence *= _CODE_EXAMPLE_DOWNWEIGHT for a finding whose
    context reads as a code example. --no-llm only.
  • meta_analyzer.py:546-566 — the LLM enrichment branch sets message=expl, confidence=conf,
    remediation=rem, explanation=expl.
  • meta_analyzer.py:520-523 — appends the llm-unconfirmed tag on the LLM path.

So the fingerprint depends on how the run was invoked, which is exactly what
finding_fingerprint's own docstring, docs/SUPPRESSION.md and .claude/rules/suppression.md
tell readers it never does. Those three texts have been narrowed to the enforceable form (no
analyzer may encode run configuration into a Finding) and now name this as the one known open
violation, inherited rather than introduced.

Impact

Every team that commits a baseline. A CI job following the README's own recipe generates its
baseline with --no-llm and is safe only while every consumer keeps the flag; a developer running
skillspector scan --baseline locally with credentials configured sees every accepted finding
return with its risk points. Nothing warns. The scanner-version guard in partition_findings does
not catch it — the version is identical, only the payload moved.

Not reachable through --spec-checks: that leak was closed by moving the mode into
unscored_rule_ids, and a conformance finding bypasses apply_filter unchanged
(meta_analyzer.py:480) precisely so it does not acquire this one.

Gotchas

  • Do not "fix" it by dropping the LLM-mutated fields from the payload. That re-fingerprints
    every existing v2 baseline entry in the scanner, not just the affected ones — the same objection
    A baseline is silently bound to the --spec-checks mode it was taken in; the document records no mode and scan cannot warn #122 records against dropping confidence. It is a format break, and every committed baseline
    goes stale silently, because a fingerprint that no longer matches suppresses nothing rather than
    erroring.
  • _fallback_filtered is the --no-llm path and it is not a no-op. It is easy to assume the
    flag only skips work; it also downweights, so the divergence exists even when the LLM would
    have confirmed nothing.
  • A test asserting fingerprint stability across the two paths would need a stubbed LLM, which
    tests/nodes/test_meta_analyzer.py already does — the absence of such a test is why this has
    never failed CI.
  • The --spec-checks exemption at meta_analyzer.py:480 is the one place the codebase already
    reasons about this leak; it is scoped to that catalogue and does not generalise.

Options

  1. Fingerprint pre-enrichment. Carry the pre-meta-analysis fingerprint on the finding (or
    re-derive it in report from the raw findings) and match baselines against that. Bounded, and
    it makes the invariant true rather than narrowed. Cost: a second fingerprint concept, and the
    baseline document's existing hashes still move once.
  2. Exclude the five LLM-mutated fields from the payload. Smallest code change, largest blast
    radius: every existing v2 baseline breaks, and the fingerprint loses the enrichment evidence it
    was widened to include.
  3. Record the binding in the baseline document and warn on mismatch. Same shape as A baseline is silently bound to the --spec-checks mode it was taken in; the document records no mode and scan cannot warn #122's
    options: baseline writes which side of --no-llm it was generated on, scan warns when they
    disagree. Does not fix the hashing; makes it visible at the moment it matters.
  4. Document only. The current state after this change: the texts name the violation and link
    here.

Start with 3 — it is the disclosure gap that actually harms users, it composes with #122's fix
(both are "the document records no mode"), and it does not invalidate a single committed baseline.
1 is the real repair and should be measured before it is chosen.

Activity

  1. rodrigorjsf commented on Aug 22, 2026

    @rodrigorjsf
    rodrigorjsf
    on Aug 22, 2026
    OwnerAuthor

    This was generated by AI during triage.

    Agent Brief

    Category: bug
    Summary: Disclose the --no-llm binding of a v2 baseline fingerprint — record the mode in the baseline document and warn when a scan disagrees with it. Option 3 only.

    Verification. The mechanism was re-confirmed against the current tree during triage:
    finding_fingerprint's payload hashes confidence, message, remediation, explanation
    and tags, and its own docstring states the invariant it is violating ("no analyzer may
    encode how the run was configured into a Finding"). The meta-analysis stage writes every
    one of those names on one branch or the other, and which branch runs is decided by --no-llm.
    The issue's reproduction stands.

    Current behavior:
    A baseline generated with --no-llm suppresses nothing on a scan run without it, and the
    reverse. Nothing warns; the scanner-version guard does not catch it because the version is
    identical and only the payload moved. README.md recommends the --no-llm shape for CI and
    docs/SUPPRESSION.md documents the baseline subcommand's --no-llm flag with no caveat.

    Desired behavior:
    The baseline document records which side of the LLM boundary it was generated on. A scan that
    loads a baseline generated on the other side emits a warning naming the mismatch and what it
    means (fingerprints will not match; accepted findings will return). The scan still runs, still
    exits on its own rules, and suppresses whatever genuinely matches. No existing committed
    baseline is invalidated     — a document without the new key is read exactly as it is read today.

    Key interfaces:

    • The baseline document's top-level metadata block — a new key naming the LLM/meta-analysis mode
      the document was generated under. Must be optional on read: absent means "unknown", which
      warns nothing.
    • The scan-side baseline loader — compares the document's recorded mode against the current
      run's and emits the warning on disagreement.
    • The warning goes to stderr, never to a piped report, for the same reason the
      Spec conformance: line does.
    • finding_fingerprint's payload is not touched.

    Acceptance criteria:

    • A baseline written today records the mode it was generated under.
    • A baseline document written before this change (no mode key) loads without error and
      without a warning.
    • Loading a --no-llm-generated baseline into a run with LLM enrichment active warns on
      stderr, naming both sides; the reverse case warns too.
    • Matching modes emit no warning.
    • The warning is on stderr and absent from stdout under every --format. Assert
      result.stdout and result.stderr apart — result.output folds both streams.
    • Every committed baseline fixture in the repository still suppresses exactly what it
      suppressed before the change.
    • docs/SUPPRESSION.md, .claude/rules/suppression.md and the README's baseline/CI
      recommendation state the binding and the new key; the README's Contributing table row
      for suppression/baseline is updated in the same PR.
    • make test-unit, make lint, make format-check pass, and make update-snapshots
      leaves the tree clean.

    Out of scope — do not do any of these:

    • Option 1 (fingerprint pre-enrichment) and option 2 (drop the five LLM-mutated fields).
      Both are baseline-format breaks. A fingerprint that no longer matches suppresses nothing
      rather than erroring      , so every committed baseline in every consuming project goes stale
      silently. If the work suggests option 1 is the right repair, say so in the PR body and file
      it as its own issue — do not take it here.
    • Changing finding_fingerprint's hashed field set for any reason.
    • #122's --spec-checks mode disclosure. It is the same shape of fix and it composes, but
      it is a separate issue and a separate PR.
    • Removing the --spec-checks exemption in the meta-analysis stage.
  2. rodrigorjsf
    added
    bugSomething isn't working
    ready-for-agentAgent: ready for Claude to pick up and work on
    on Aug 22, 2026

  3. rodrigorjsf commented on Oct 10, 2026

    @rodrigorjsf
    rodrigorjsf
    on Oct 10, 2026
    OwnerAuthor

    Surfaced by the 2.12.0 upstream sync (#142), not changed by it.

    The count this issue rests on — --no-llm moves five hashed fields (confidence, message, remediation, explanation, tags) — may no longer be the whole set. Upstream's meta stage now also writes code_snippet=f.code_snippet or f.context in apply_filter's enrichment branch, in _mark_unconfirmed and in _mark_review_failed (src/skillspector/nodes/meta_analyzer.py), and suppression.finding_fingerprint hashes code_snippet.

    Not measured: whether the --no-llm path (_fallback_filtered) fills it the same way, in which case the field does not move with the flag and the count stands. That measurement decides between five and six.

    The claim is written in README.md, docs/SUPPRESSION.md, docs/MULTI_FRAMEWORK_SKILL_ANALYSIS.md, CLAUDE.md (Applied Learning) and the finding_fingerprint docstring; all five need the same answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-agentAgent: ready for Claude to pick up and work on

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions