Vision aggregate envelope: parameterize (or raise) the 32,768 merged-token prompt ceiling for multi-image document workloads

[ArietidsZ]

Summary

The aggregate Vision budget is a compiled constant, kMaximumPromptVisionTokens = 32'768
merged (131,072 raw) in targets/qwen3_6/export/.../prepared_prompt.h:24, and the only
aggregate enforcement site is frontend.cpp:854-857:

const std::uint64_t vision_tokens =
    std::min<std::uint64_t>(options.max_context, kMaximumPromptVisionTokens);
processor.max_vision_tokens = vision_tokens;
processor.max_raw_patches   = vision_tokens * kRawPatchesPerVisionToken;

With a large --max-context, this constant — not the context window — becomes the binding
constraint for the Vision share of any request. We would like to ask whether it can be
parameterized (or raised). One thing up front: today's aggregate equals the registered
envelope maximum, so a flag that only lowers stays inside the §2.5 domain — but raising
past 32,768 merged exceeds the registered envelope. Option A below is the mechanism;
whether the envelope itself may grow is Question 3, and we treat that as the real decision.

Real workload (measured)

Agentic document QA on full-page scans (2264×3200 → 2272×3200 after 32-px alignment ≈
7,100 merged tokens / 28,400 raw patches each):

• 1–2 pages per request: HTTP 200 (measured)
• 6 pages per request: HTTP 400 media_budget_exceeded — "vision raw patches exceed
  processor budget" (processor.cpp:634) (measured)
• 3–5 pages: bounded by the same arithmetic (5 × 28,400 = 142,000 > 131,072)

Our production request logs show the same rejection on real traffic, not just probes:

• a compaction-replay request (189 messages, 14 media items — ≈ 3× the raw-patch budget
  if the items are page scans of this size) → 400 media_budget_exceeded, twice;
• single-turn requests carrying 6 and 12 page images → 400, same error.

Long agent sessions that re-send images during compaction replays hit this repeatedly,
and the client-side workaround (pre-scaling pages down) costs OCR fidelity on dense text —
a ~50% linear downscale still leaves 14 pages over the envelope, and the ~70% needed to
fit is a severe quality loss. This is exactly the trade an operator would prefer to make
consciously.

Why raising the aggregate looks cheap (source audit @ 36f23d7)

1. Device work does not scale with the aggregate — structurally. The source comment
   (prepared_prompt.h:22-23) is explicit: aggregate capacity and one-item execution
   capacity are intentionally distinct; items pass through the Vision tower sequentially.
   The workspace planner clamps vision per item (layouts_impl.h:581,
   min(plan.capacity, kMaximumVisionItemTokens)), bind_output rejects per-item
   overruns, and the handoff is one fixed [5120,V] allocation with
   V ≤ min(max_context, 16384) per item, reused across items. The split is visible in
   the memory summary itself: aggregate_prompt_tokens is recorded as metadata while
   encode_peak_bytes/handoff_* all derive from max_item_tokens
   (program_impl.h:11526-11537). We could not find any device allocation, reservation,
   plan dimension, or rejection path that scales with the aggregate — if one exists, we
   would be glad to be corrected.
2. Host-side retention is guarded — twice. frontend.cpp:859-867 refuses startup
   unless --media-live-mib holds max_raw_patches × 1536 × 2 B (3 KiB per raw patch),
   and the same value seeds the request-level preparation gate
   (floor(live / per-request-max)), which queues excess concurrent vision preparations
   (HTTP 504 on timeout) instead of over-committing. Note the defaults are co-sized for
   today's envelope (2048 MiB ≈ 5 × 384 MiB): raising the aggregate without raising the
   flag silently throttles vision concurrency to 2 lanes at 65,536 merged, 1 lane at
   131,072. Sizing: --media-live-mib ≥ concurrency × raw × 3 KiB + cache residency;
   identical media across concurrent requests (compaction replays) share one payload via
   the cache and count once.
3. KV draw is the operator's own knob. Merged tokens occupy the KV pool like text —
   fp8 ≈ 32.25 KiB/token here (16 full-attention layers × 4 KV heads × head_dim 256,
   including scale pages) — a trade against --kv-capacity, not an engine cost.
4. Per-item grids are unchanged by an aggregate raise. Each item still caps at 16,384
   merged / 65,536 raw (exactly the artifact's preprocessor_config.json capability:
   longest_edge 16,777,216 px), and the 48×48 learned position-table interpolation is
   computed from each item's own grid (impl/vision/control.cpp, kPositionSide = 48) —
   never from the aggregate sequence. Per-item processing is therefore bit-for-bit
   identical. Whether overall output quality changes when more items share one prompt
   (more vision tokens in text-prefill attention) is a model-side question we cannot
   assess from source — hence Question 3.

Memory math on RTX 5090 32 GB, qwen3.8-27b groupwise-int, fp8 KV, --max-context 262144,
--kv-capacity 262144 (weights 16.67 GiB, KV runtime 10.89 GiB, free-after-startup
2.36 GiB, from startup logs):

Aggregate envelope	host live, per vision request	device	KV pool draw
32,768 merged (today)	384 MiB	unchanged	~1 GiB
65,536 merged	768 MiB	unchanged	~2 GiB
131,072 merged	1.5 GiB	unchanged	~4 GiB

Provenance of the 32,768 (why we ask)

• The per-item bound is capability-derived: 65,536 raw = preprocessor_config.json
  longest_edge ÷ 16² px per patch; 16,384 merged = ÷ 4. The code enforces these pixel
  bounds as a registration contract (frontend.cpp:192-201). The aggregate bound is not
  registration-enforced anywhere — it is the frontend default.
• The aggregate bound is exactly 2× the per-item bound. It has been present since the
  native-target cutover (0aedcb0, 2026-07-14, a ~300-file batch import) and moved into
  the shared family header by 6302f22. We could not find a commit message or doc that
  justifies the factor 2 — full-repo search over commit messages, docs, and issues. We
  may well be missing a reason; hence the questions below.
• It has survived three budget refactors alongside two budgets upstream has since
  removed (0c94153): max_attention_pairs, which fix(vision): remove the cumulative attention_pairs budget #19 showed contradicted the artifact by
  5.4× in pixels / 32× in pairs, and max_prompt_tokens, which was subsumed by
  --max-context. We cite these as provenance, not precedent — the aggregate differs
  from both: unlike max_attention_pairs it bounds a real reserved resource (media-live
  bytes), and unlike max_prompt_tokens nothing larger supersedes it.
• docs/maintainer/qwen3.6-27b-model.md §2.5 registers 4≤P≤131072, 1≤V≤32768 as the
  implementation envelope and says frontend budgets impose "any additional
  request-specific restriction" — today's frontend does not expose the aggregate as a
  request- or server-level budget at all; it hard-wires the registered domain as the only
  ceiling.

Proposal

Option A (preferred): expose the aggregate ceiling as a serve flag with today's
default, e.g. --vision-max-prompt-merged N (default 32768, bit-for-bit unchanged):
thread it ServeOptions → EngineOptions → FrontendOptions and use
min(max_context, N) at frontend.cpp:855; keep per-item caps untouched so single-item
behavior, the sequential tower contract, and the registered pixel-bounds registration
check are unchanged for default deployments. We estimate ~60–90 lines across ~10–12 files
including the program_impl.h:11529 telemetry clamp (otherwise it underreports), tests
(test_load_plan.cpp currently pins the constant with a static_assert), and the three
docs. The existing media-live startup check plus the preparation gate then guard both the
memory and the concurrency cost automatically.

Option B (complementary UX): #72's fit-and-scale is per-item — for pages of our size
(7,100 merged < 16,384 per-item cap) it never triggers, and the aggregate ceiling
(kFrontendMergedLimit = 32768 in layouts_impl.h) remains binding after #72 merges: a
5-page request still exceeds 131,072 raw. When the aggregate budget is the constraint,
the request is rejected outright — consider an optional aggregate-level fit-and-scale, or
include in the 400 payload which budget tripped and the scale factor that would fit.

Alternatives we considered and why they don't fit: client-side OCR of pages trades away
layout/stamps/figures and still fills --max-context; batching into multiple requests
doesn't help compaction replays, which re-send the full history; deriving the envelope
from --media-live-mib conflates a host retention buffer with a processing domain.

Questions

1. Is there an implementation-domain reason for 131,072 raw / 32,768 merged that we
   missed (anything on the device that scales with the aggregate, or a checkpoint-side
   constraint) — or is 2× the intended headroom for "two maximum-size images" (as fix(vision): remove the cumulative attention_pairs budget #19
   described it)?
2. Would you rather fold this into On-demand vision residency: stream the tower through evicted read-only text weights #72's --vision-max-merged (per-item; its [64, 32768]
   clamp cannot raise the ceiling), or keep it a separate flag?
3. Is a registered-envelope change (values above 131,072 raw) something you would
   consider for this target at all, or is the aggregate bound a quality-motivated choice
   (e.g., total interpolated position rows per prompt)?

Environment

Neroued/ninfer master 36f23d7, ninfer-serve, RTX 5090 32 GB, qwen3.8-27b groupwise-int
artifact, --kv-dtype fp8 --max-context 262144 --vision --spec mtp. Related: #72, #74,
#61, #19, #20.