Recently, there have been several discussions and implementations around prefix caching, Host KV, checkpoint retention, and resource eviction. I want to give a unified update on NInfer’s current direction and use it to resolve the related Issues and PRs.
First, to clarify the timeline: I had already listed “better prefix caching” as planned work in TODO List #36. In #33, I also explicitly described the old prefix-caching behavior as a limitation of the engine design at that time and said that I planned to introduce a more general design. The resource scheduling architecture now in master is the completion of that existing plan. Later Issues and PRs contributed useful workloads, measurements, and failure cases, but they are not the source of the current architecture.
The original problem was clear: with limited VRAM, a short request could displace a long-running session, while multiple agents alternating or branching from a shared prefix could easily turn simple lane retention into repeated full prefills.
Rather than adding another cache layer, I rebuilt admission around complete, exact checkpoints and a single Program-owned Device/Host resource model. A checkpoint is reusable only when the target can restore all State and typed KV required to continue, while active requests reserve the capacity they need to finish.
The common no-pressure case stays on a direct path. Under pressure, the planner compares feasible outcomes across root, private, and shared checkpoints, accounting for transfers, remaining prefill, and the future value lost by degrading other cached contexts. Program then applies the selected placement through its own physical stores, so reuse, Host restore, demotion, and eviction no longer depend on competing cache owners.
This model covers restoring a long request after a short request displaced it, alternating sessions, shared system/tool prefixes, trajectory branching, rewritten context, and long tool loops. Compared with adding an LRU, whole-lane slabs, or a content-addressed store in isolation, the important advantage is that every case follows the same resource invariants. Aliasing, temporary copy peaks, and incomplete backend state cannot be hidden behind an apparently feasible cache decision.
The detailed design is documented here:
Related discussions include #40, #49, #51, #62, and #75. Related implementations include #64, #73, and #90:
Thank you for the real workloads and measurements contributed through these discussions. The architecture upgrade that was previously planned is now implemented, so I will close the related Issues and PRs that have been superseded or fall outside the project’s direction. I do not plan to continue iterating on the old implementations or merge them after further refactoring.
Recently, there have been several discussions and implementations around prefix caching, Host KV, checkpoint retention, and resource eviction. I want to give a unified update on NInfer’s current direction and use it to resolve the related Issues and PRs.
First, to clarify the timeline: I had already listed “better prefix caching” as planned work in TODO List #36. In #33, I also explicitly described the old prefix-caching behavior as a limitation of the engine design at that time and said that I planned to introduce a more general design. The resource scheduling architecture now in master is the completion of that existing plan. Later Issues and PRs contributed useful workloads, measurements, and failure cases, but they are not the source of the current architecture.
The original problem was clear: with limited VRAM, a short request could displace a long-running session, while multiple agents alternating or branching from a shared prefix could easily turn simple lane retention into repeated full prefills.
Rather than adding another cache layer, I rebuilt admission around complete, exact checkpoints and a single Program-owned Device/Host resource model. A checkpoint is reusable only when the target can restore all State and typed KV required to continue, while active requests reserve the capacity they need to finish.
The common no-pressure case stays on a direct path. Under pressure, the planner compares feasible outcomes across root, private, and shared checkpoints, accounting for transfers, remaining prefill, and the future value lost by degrading other cached contexts. Program then applies the selected placement through its own physical stores, so reuse, Host restore, demotion, and eviction no longer depend on competing cache owners.
This model covers restoring a long request after a short request displaced it, alternating sessions, shared system/tool prefixes, trajectory branching, rewritten context, and long tool loops. Compared with adding an LRU, whole-lane slabs, or a content-addressed store in isolation, the important advantage is that every case follows the same resource invariants. Aliasing, temporary copy peaks, and incomplete backend state cannot be hidden behind an apparently feasible cache decision.
The detailed design is documented here:
Related discussions include #40, #49, #51, #62, and #75. Related implementations include #64, #73, and #90:
SharedStablePrefixand the unified Device/Host planner. That PR also introduces a third home for sequence state and includes a large amount of unrelated work, so it will not be adopted.Thank you for the real workloads and measurements contributed through these discussions. The architecture upgrade that was previously planned is now implemented, so I will close the related Issues and PRs that have been superseded or fall outside the project’s direction. I do not plan to continue iterating on the old implementations or merge them after further refactoring.