apr chat silently loads the toy Demo model for sharded SafeTensors (.safetensors.index.json) instead of the real model

[alfredodeza]

Summary

apr chat silently substitutes the built-in toy demo model for any sharded
SafeTensors model — with no error, no warning, and the banner still shows the
real model's path — instead of loading the actual model or refusing clearly.
This breaks the exact workflow apr pull itself recommends for any 3B+
SafeTensors model.

Repro

apr pull hf://Qwen/Qwen2.5-7B-Instruct
# ...
# ✓ Downloaded successfully
#   Path: /home/alfredo/.apr/cache/hf/Qwen/Qwen2.5-7B-Instruct/model.safetensors.index.json
#   Shards: 4
#
#   Usage:
#     apr run /home/.../model.safetensors.index.json
#     apr serve /home/.../model.safetensors.index.json

apr chat /home/alfredo/.apr/cache/hf/Qwen/Qwen2.5-7B-Instruct/model.safetensors.index.json --gpu

Output:

=== Chat Demo (Tiny Model) ===

Note: Using tiny demo model. Pass .apr, .gguf, or .safetensors file for full model.

  Model: /home/alfredo/.apr/cache/hf/Qwen/Qwen2.5-7B-Instruct/model.safetensors.index.json
  Chat Template: Raw
  ...
Loading model...
Loaded Demo format in 0.00s (0.0 MB)
Warning: Could not detect chat template for 'demo', using raw format (no ChatML/Instruct wrapping)
You: hey
[0 tokens in 0.0s = 0.0 tok/s]
Assistant:

You:

No model was loaded (0.0 MB, 0 tokens generated). The banner still prints the
real model's path right above "Chat Demo (Tiny Model)", which makes it look
like the 7B model loaded and produced an empty/garbage response — it never
touched the real model at all.

Root cause

crates/apr-cli/src/commands/chat.rs::detect_format (line ~214):

fn detect_format(path: &Path) -> ModelFormat {
    match path.extension().and_then(|e| e.to_str()) {
        Some("apr") => ModelFormat::Apr,
        Some("gguf") => ModelFormat::Gguf,
        Some("safetensors") => ModelFormat::SafeTensors,
        _ => ModelFormat::Demo,
    }
}

Path::extension() on model.safetensors.index.json returns Some("json")
(the last dot-segment), which matches none of the three arms, so it falls
through to ModelFormat::Demo.

The secondary magic-byte detector, detect_format_from_bytes (same file,
#[cfg(feature = "inference")]), can't rescue this either: the file at that
path is a JSON index manifest, not tensor binary data, so its header-size
heuristic also can't recognize it as SafeTensors.

Compare to apr run's inference path
(crates/aprender-serve/src/infer/inference_result.rs::run_inference), which
DOES special-case this:

let path_str = config.model_path.to_string_lossy();
if path_str.ends_with(".safetensors.index.json") {
    ...
    return run_sharded_safetensors_inference(config, &prepared);
}

apr chat has no equivalent branch anywhere in commands/chat.rs — no
ShardedSafeTensorsModel, no .index.json suffix check, nothing. Sharded
SafeTensors support exists in the codebase (crates/aprender-core/src/format/sharded_index.rs,
crates/aprender-serve/src/safetensors_infer.rs) but apr chat never wires
into it.

Impact

• Any model 3B+ pulled as SafeTensors (i.e. anything apr pull shards, which
  is apr pull's own default for models of that size) cannot be used with
  apr chat at all — it silently runs the toy demo instead.
• No error is raised, so this fails quietly. A user unfamiliar with the
  internals reasonably concludes their large model produced an empty/garbage
  response, or that --gpu broke something, rather than realizing their real
  model was never loaded.
• apr pull's own printed instructions recommend apr run/apr serve for
  this exact path but do NOT mention apr chat — so the fix might also be:
  make apr chat on an unsupported format at minimum print the same
  "Note: Using tiny demo model" Demo fallback loudly enough (e.g. as an
  actual error:/refusal) that it can't be mistaken for real model output,
  until the sharded-safetensors path is wired in.

Ask

1. Wire apr chat into the existing sharded-SafeTensors inference path
   (ShardedSafeTensorsModel / run_sharded_safetensors_inference), the same
   way apr run already is, so .safetensors.index.json loads the real model.
2. Until that lands, make the Demo fallback fail loudly instead of silently
   when the input path exists and does not match any real extension — a
   demo-model substitution for an existing, non-trivial file the user
   explicitly pointed at should never look like a normal successful load.

Environment

• apr 0.65.2 (b1a6324b8)
• Model: Qwen/Qwen2.5-7B-Instruct, sharded SafeTensors (4 shards + index.json),
  pulled via apr pull hf://Qwen/Qwen2.5-7B-Instruct

[noahgift]

Reproduced, root-caused and fixed — PR #3050. Brought into the 0.66 release (milestone 0.66.0, P0, PP-066 row F-1, ticket PMAT-1080).

Reproduced on origin/main, offline

apr 0.65.2 (c04eda87d) built from the tip of main, against a 2-shard fixture derived from a model already on the box (scripts/make_sharded_safetensors.py, no download):

=== Chat Demo (Tiny Model) ===
  Model: .../sharded-fixture/model.safetensors.index.json
Loaded Demo format in 0.00s (0.0 MB)
You: [0 tokens in 0.0s = 0.0 tok/s]
$ echo $?   ->  0

Your diagnosis of the mechanism is exactly right, and the detect_format line you quoted is the line.

One correction to the report

apr pull's own printed instructions recommend apr run/apr serve for this exact path

Measured here, apr run on the sharded index works — Output: 2 + 2 equals 4., exit 0, on unmodified main. So the sharded inference path is fine; the defect is apr chat alone, which never wires into it. If your apr run on the real 7B pull did fail, that is a different defect — please paste the exact command and output and I will open it separately; it is not reproduced by this fixture and stays [U]. The new gate now covers the sharded × apr run cell on every run, so a regression there would be caught rather than inferred.

The fix takes ask 2 as the general rule, not the fallback

Your ask 2 ("until that lands, make the Demo fallback fail loudly") turned out to be the right permanent rule, so it is not a stopgap here. resolve_chat_model already proves the file exists before anything else runs — so every Demo in apr chat was a silent substitution for a real file the user named. Demo is now not an outcome for an existing path at all:

$ apr chat /tmp/notes.txt
error: Model load failed: /tmp/notes.txt is not a model format apr chat can load.
  Recognised: .apr, .gguf, .safetensors, and a sharded SafeTensors index (*.safetensors.index.json).
  apr chat will not substitute its built-in demo model for a file you named (#3022).
$ echo $?   ->  6      # CliError::ModelLoadFailed, read from error.rs, never typed

And ask 1 lands with it — .safetensors.index.json now reaches the transformer through the same three calls apr run already makes (load_from_index → SafetensorsConfig::load_from_sibling → convert_sharded):

=== Model Chat (Sharded SafeTensors) ===
Loaded Sharded SafeTensors format in 0.00s (988.1 MB)
Loaded config: 24 layers, 896 hidden, 14 heads
Detected ChatML chat template
You: [12 tokens in 3.7s = 3.2 tok/s]
Assistant: 2 + 2 equals 4.

Why one added arm was not the fix

The extension bug is the instance. The class is that two independent decisions existed — print_welcome_banner asked the PATH, ChatSession::new asked the first eight BYTES, and nothing compared them. "Banner says SafeTensors, session loads Demo" was a reachable state with no falsifier anywhere in the tree, which is why it shipped. There is now one decision (resolve_chat_format: suffix before extension(), then magic bytes, then refuse) and every consumer is handed its single answer.

Two further honesty fixes fell out: the load line reports the manifest's metadata.total_size for a sharded model (printing the ~20 KB index itself would have been the same lie under a different name — 0.0 MB), and an odd-named file with real magic bytes still resolves to its true format, so the refusal cannot become a false one.

Falsifiers: 6 unit rows (both polarities) in workspace-test, a 7-row judge case table and a 7-row splitter case table in ci / gate, contract contracts/patterns/format-command-honesty-v1.yaml. Both registered mutants turn the table RED. Receipt: docs/audits/impl-PMAT-1080-receipt.md.