roadmap.md

Roadmap

Rules for every milestone

A milestone is complete only when its acceptance criteria, correctness evidence, and required measurements are recorded. Performance work must state a hypothesis, compare a baseline on the same hardware, and preserve deterministic correctness. GPU results record GPU, driver, CUDA, model and revision, dtype, input/output lengths, batch or concurrency configuration, commit hash, and build features.

Milestone 0: Repository foundation

Goal: Establish a maintainable project without premature crate or trait boundaries.

Dependencies: Stable Rust; no GPU required.

Deliverables:

• one Cargo package, typed CLI, typed errors, structured tracing, and feature-gated CUDA;
• architecture, roadmap, dependency policy, benchmark instructions, README, and contribution guide;
• CI for format, build, test, and Clippy;
• local/Hub artifact resolution, sharded index discovery, config parsing, tokenizer validation, and an explicit model registry;
• unresolved decisions listed in the architecture document.

Acceptance: cargo build --locked, cargo test --locked --all-targets, cargo fmt --all --check, and strict Clippy pass on a non-CUDA host. The inspect command recognizes a valid local fixture and rejects unsupported architectures or missing/unsafe shard paths.

Benchmark deliverable: Define the result schema and measurement methodology; do not add meaningless microbenchmarks.

Milestone 1: Single-model end-to-end inference

Status: CPU/F32 complete (July 2026). CUDA/BF16 runtime and benchmark validation pass on RTX 5090; exact PyTorch CUDA/BF16 greedy parity is pending.

Selected model: Qwen3-0.6B (Qwen/Qwen3-0.6B, commit c1899de289a04d12100db370d81485cdf75e47ca), BF16 dtype at the checkpoint.

Deliverables:

• typed Qwen3Config with semantic validation for architecture, dimensions, GQA ratios, head_dim, dtypes, RoPE scaling, sliding window, token-ID bounds);
• raw-tokenizer encoding with add_special_tokens=false;
• SafeTensors full-buffer loading and validation: 311 BF16 tensors, exact names and shapes, tied-weight verification, and an owned Vec<u8> before device transfer;
• Candle primitives: RMSNorm, precomputed RoPE, GQA attention, SwiGLU MLP;
• 28-layer decoder with pre-norm residual structure and causal masked attention;
• preallocated contiguous per-layer KV cache ([1, Hkv, capacity, Dh]), BF16 or F32, allocated once per request;
• cached decode loop using the same forward_cached path as prefill;
• generate CLI command: --model, --prompt, --max-new-tokens, --device, --dtype;
• timing metrics: model_load_ms, prefill_ms, ttft_ms, mean_tpot_ms, decode_tokens_per_sec, resolved device and dtype;
• BF16 preflight probe on selected device before weight loading;
• eager-attention operational prefill cap of 2,048 prompt tokens, separate from the model's 40,960-position semantic limit;
• EOS token IDs loaded from generation_config.json (falls back to config.json);
• deterministic Python reference oracle (tools/reference_qwen3.py) with pinned dependency versions and SHA-256 model identity validation.

Acceptance:

• cargo run -- generate --model models/Qwen3-0.6B --prompt "Hello" prints a decoded continuation on stdout and metrics on stderr;
• CPU/F32 tokenizer IDs, selected tensors, decoded text, and exact 32-token greedy output match the pinned reference fixture;
• cached decode matches full-context recomputation token-by-token;
• all quality gates pass on a non-CUDA host;
• CUDA/BF16 passes a preflight probe, generation smoke test, and recorded benchmark on RTX 5090;
• CUDA/BF16 is not considered numerically complete until its first divergence from the matching PyTorch oracle is localized and resolved or accepted through a documented backend-specific correctness policy.

Milestone 2: Non-HTTP engine

Status: Complete (July 2026). See docs/milestone-2-report.md.

Goal: Separate model execution from request and sequence lifecycle.

Dependencies: Correct Milestone 1 generation path.

Implemented deliverables:

• a public library API with EngineConfig, Engine, GenerationRequest, GenerationHandle, request IDs, greedy SamplingParams, token events, terminal summaries/failures, and classified finish reasons;
• a named dedicated model-runner thread with a capacity-one readiness handshake;
• one-time worker ownership of the tokenizer, Qwen3 model, device/dtype, EOS configuration, and startup metrics;
• a bounded capacity-eight Crossbeam command queue and bounded per-request event channels sized for all requested tokens plus one terminal event;
• serialized FIFO request execution: one active sequence, no batching or interleaving, with multiple sequential requests reusing the loaded model;
• a concrete worker-private ModelRunner that owns Candle execution and exposes cache allocation, prefill, decode, and synchronization without CLI/channel dependencies;
• a worker-private Sequence state machine with checked num_computed_tokens, cache-length equality, and a fresh contiguous cache per request;
• tokenizer-safe incremental text using DecodeStream, plus final trailing-text reconciliation against a complete decode;
• cooperative cancellation through GenerationHandle, handle-drop cancellation, receiver-drop detection, explicit/idempotent shutdown, queued request draining, and worker joining;
• startup, submission, recoverable request, fatal worker, and panic failure paths;
• worker-owned queue, tokenization, prefill, TTFT, per-step decode, TPOT, throughput, and total-generation metrics, with model-load time separated into startup metrics;
• a thin CLI that only starts the engine, submits, consumes events, prints the returned text/metrics, and shuts down.

Acceptance:

• the CLI has no direct tokenizer/model/cache/sampling calls;
• the model runner contains no CLI, channel, request-ID, or future HTTP types;
• public API integration tests preserve the canonical 32-token CPU/F32 output, exercise two sequential requests, reconstruct final text from streamed fragments, cancel after a token, and successfully execute a later request;
• sequence tests enforce num_computed_tokens <= total tokens and cache length equality, including stale-cache rejection;
• accepted attached requests emit ordered tokens and exactly one terminal event;
• shutdown rejects new submissions, cancels active/queued requests at safe boundaries, and joins the worker;
• all required repository gates and local ignored tests pass.

Benchmarks: The fake worker benchmark delivered 320,000 token events across 10,000 requests and measured first-event, buffered per-token/terminal receive, shutdown/join, and queue-overload behavior. The CPU/F32 regression used one warmup and five candidate samples against the retained Milestone 1 artifact. Median TTFT was 68.3 ms versus 75.0 ms; median TPOT was 69.3 ms versus 74.2 ms. Both pass the max(5%, 1 ms) regression threshold. The retained baseline has one sample, so baseline spread and true interleaving could not be reconstructed.

Milestone 3: Streaming HTTP serving

Status: Complete (July 2026). See docs/milestone-3-report.md.

Goal: Expose the engine through an OpenAI-compatible network boundary.

Dependencies: Milestone 2 completion gates and a stable submit/event/cancel/shutdown lifecycle.

Implemented deliverables:

• Tokio/Axum server with GET /health, GET /ready, GET /v1/models, POST /v1/completions, and POST /v1/chat/completions;
• strict OpenAI-shaped request/response DTOs, model identity, usage accounting, public request IDs, and stable structured public errors;
• streaming completion/chat SSE chunks with one terminal [DONE], and matching non-streaming responses;
• one finite spawn_blocking event consumer per accepted generation, forwarding into a bounded Tokio channel without moving model execution off its existing dedicated worker;
• bounded HTTP admission, explicit 429 overload, slow-consumer timeout, disconnect cancellation, task tracking, and no unbounded response buffer;
• liveness separated from readiness, with readiness reading maintained engine and admission state without model execution;
• Qwen3 tokenizer_config.json chat-template compilation, text-only ordered roles, assistant generation prompt, and thinking disabled by default;
• SIGINT/SIGTERM graceful shutdown that stops admission, cancels bridge and engine work cooperatively, joins the worker, and awaits tracked bridge tasks;
• local directory or Hugging Face model source configuration, with synchronous Hub resolution during worker startup;
• documented compatibility subset that rejects sampling controls, stop, tools, multimodal content, multiple choices, prompt arrays, logprobs, response-format controls, and unknown fields.

Acceptance:

• fake-backend tests prove handlers use the engine boundary, structured errors are stable, health/model routes never submit work, queue full maps to 429, streams are ordered and terminate once, body limits are enforced, and readiness observes fatal worker state;
• bridge tests prove ordered bounded forwarding, terminal-event finality, receiver-drop cancellation, slow-consumer cancellation, and tracked cleanup;
• ignored local CPU/F32 HTTP tests verify completion/chat stream versus non-stream text equality, the canonical direct Engine 32-token ID sequence, [DONE], disconnect recovery, and graceful process shutdown with the pinned Qwen3 model;
• fake HTTP collector tests record the exact ordered IDs consumed by JSON and SSE and verify both translate Hello, 32 tokens, and greedy sampling identically;
• no Axum handler owns or invokes the tokenizer, Qwen3 model, Candle tensor, KV cache, device, or model runner;
• shutdown closes admission before engine cancellation and joins all owned worker/bridge tasks.

Benchmarks: The published v2 fake artifact records equivalent prebuilt and full-handler JSON/SSE paths, direct bounded-bridge forwarding, 32-event slow-consumer saturation, concurrency 1/8/32, cleanup latency, RSS, capacities, and correctness. The canonical real CPU/F32 artifact uses prompt Hello and 32 tokens, reports fresh and verified persistent TCP separately, and defines TTFT, TPOT, and output throughput at content-token boundaries. The two result classes remain separate because the fake path excludes model, tokenizer, engine queue, proxy, and external network costs; it includes a separate interleaved loopback TCP health workload to isolate connection/request overhead. See docs/milestone-3-report.md.

Milestone 4: Concurrent scheduling and continuous batching

Status: Complete for serial one-shot prefill and variable-length batched decode with contiguous per-sequence caches. CPU/F32 correctness and a paired same-host CUDA/BF16 serial-versus-batched benchmark are recorded in the Milestone 4 report.

Goal: Execute multiple decode sequences in one model invocation and change active membership between iterations.

Dependencies: Engine ownership and stable single-sequence cache semantics.

Deliverables: Bounded FIFO pending admission, stable sequence identity, round-robin active selection, serial one-shot prefill, batched decode, variable context lengths, per-sequence positions and contiguous caches, valid-key masks, batched greedy selection, per-sequence streaming/stopping, and cancellation cleanup.

Acceptance: Batched CPU/F32 logits and exact greedy IDs match independent execution; batch size one preserves the prior path; concurrent engine requests retain independent ordered streams; one sequence finishing or cancelling does not corrupt others; fake scheduler tests prove batches larger than one and new admission while older sequences remain active.

Benchmarks: The initial batch-1/2/4 sweep at concurrency four records 33.6% and 109.8% aggregate throughput improvements for batch 2 and 4 respectively. Batch 2 improves TTFT and p95 completion latency but worsens per-request TPOT and median completion latency; batch 4 improves throughput and both completion latency summaries in this homogeneous workload. Continue mixed-length sweeps as the cache path evolves.

Milestone 5: Continuous-batching follow-on

The core waiting/active lifecycle and continuous admission originally planned here were implemented in Milestone 4. There is no separate Milestone 5 code phase. Token-budget scheduling, mixed chunked prefill/decode, and bounded long-prompt interference remain part of the later chunked-prefill milestone.

Benchmark follow-up: Retain the concurrency sweep, burst/steady arrivals, aggregate tokens/s, queue delay, TTFT/TPOT, tail latency, batch distribution, and serial-baseline comparison as ongoing performance evidence.

Milestone 6: Paged KV cache

Status: Complete (July 2026). The default is a Candle paged-reference path; the contiguous path remains selectable for differential tests and benchmarks.

Goal: Replace fixed maximum-context allocations with global physical blocks.

Dependencies: Stable sequence lifecycle and attention metadata; chosen block layout based on access measurements.

Deliverables: Global block pool, allocator, per-sequence block tables, allocation/release/rollback, mapped K/V writes and reads, memory accounting, and deterministic cleanup for finish/cancel/error. CPU allocator logic is isolated from GPU tensor storage for exhaustive testing.

Acceptance: No fixed max-context allocation per active sequence; allocator property tests cover uniqueness, reclaimability, rollback, and reference accounting; all blocks return after every lifecycle path; generated tokens match fixed-cache execution; concurrency improves under a fixed memory budget.

Benchmarks: Fixed versus paged allocation for memory utilization, maximum sequences, internal/external fragmentation, allocation overhead, attention latency, throughput, and block-size sweep.

Evidence: Pure allocator lifecycle/property tests, exact 32-token and unequal-context CPU/F32 differential tests, paged Engine cancellation/recovery tests, block-layout microbenchmarks, analytical workload accounting, and paired CUDA/BF16 HTTP artifacts are summarized in docs/milestone-6-report.md.

Milestone 7: Chunked prefill

Status: Complete (July 2026). The default scheduler uses chunked paged prefill; one-shot prefill remains selectable for differential validation.

Goal: Prevent long prompts from monopolizing scheduler iterations.

Dependencies: Continuous scheduler and a cache that safely grows by chunks.

Deliverables: Configurable per-iteration token budget and maximum prefill chunk, scheduler-level mixed decode/prefill work, correct positions/masks across chunks, decode-first fairness, full-prompt capacity guarding, and prompt-limit admission. The Candle runner uses separate decode and prefill model calls in a tick; it does not claim a fused heterogeneous batch.

Acceptance: Long prompts prefill incrementally; decode requests continue to make bounded progress; no scheduler budget violations; chunked and one-shot prefill produce identical greedy output; cancellation between chunks frees all resources.

Benchmarks: Large prompt plus concurrent decodes, chunk-size/token-budget sweep, prefill throughput, TTFT, decode TPOT and p99, GPU utilization, and comparison to unchunked continuous batching.

Evidence: Model-free scheduler tests prove shared-budget accounting and decode progress across four prompt chunks. CPU/F32 tests compare chunk sizes 1, 5, 16, and 32 across page boundaries and require exact end-to-end greedy IDs and text against one-shot execution. The workload driver and paired results are described in docs/milestone-7-report.md.

Milestone 8: Prefix caching

Status: Complete (July 2026). Prefix reuse is disabled by default and may be enabled for one trusted process-global isolation domain. See docs/milestone-8-report.md.

Goal: Safely reuse previously computed KV prefixes.

Dependencies: Immutable paged blocks, block ownership/refcounts, and stable tokenizer/model identity.

Deliverables: Typed namespace identity for model source/revision, config/checkpoint/tokenizer artifacts, dtype/backend, KV layout/page size, prompt-format policy, and isolation scope; exact cumulative token-page lookup; immutable full-page sharing; cache and sequence reference accounting; bounded LRU eviction; exact full-prompt greedy sampling state; hit/miss/reused-token/ retained-byte metrics; and a conservative privacy policy.

Acceptance: Eligible identical prefixes avoid computation; ineligible or partially matching inputs do not collide; shared blocks survive one sequence finishing and are reclaimed after final ownership/eviction; exact greedy output is unchanged.

Benchmarks: Hit-rate sweep, cached versus uncached TTFT, lookup overhead, retained memory, eviction churn, maximum concurrency, and adversarial low-hit workload.

Evidence: Pure tests cover namespace sensitivity, exact collision verification, longest-prefix lookup, duplicate insertion, LRU removal, bounded retention, shared reference lifecycles, eviction with live leases, and 10,000 randomized lifecycles. The CPU/F32 model test requires exact cached/uncached greedy IDs and text for exact and partial hits. The CUDA/BF16 paired artifact records a 254.96 ms disabled warm median TTFT versus 1.02 ms for an exact warm hit, 0/25/50/75/100% requested-prefix sweeps, concurrent shared-page streams, eviction churn, and -1.63% median TTFT on a true no-hit workload.

Milestone 9: Performance specialization

Goal: Adopt only optimizations justified by profiles and serving goals.

Candidates: FlashAttention, paged decode attention, CUDA graphs, fused RMSNorm/RoPE/QKV, pinned memory, asynchronous copies, quantization, FlashInfer integration, custom CUDA, speculative decoding, then distributed execution.

Entry criteria for each candidate: A named bottleneck from profiles; a benchmark hypothesis and baseline; isolated correctness oracle; maintainability and portability assessment; rollback path.

Acceptance: Publish the measured result with confidence/variance where appropriate, exact environment metadata, numerical error, end-to-end serving impact, and any workload regression. Feature count is not acceptance.

Cross-milestone test matrix

Unit tests cover config parsing, sampling, sequence state/cache accounting, worker command handling, event-channel invariants, request IDs, metrics, and cancellation, allocator ownership, reservation transactions, block tables, and page gather isolation. The ignored local CPU/F32 integration suite loads the pinned model, compares deterministic IDs and selected tensors, exercises streaming, sequential reuse, cancellation, recovery, contiguous-versus-paged token parity, page boundaries, and unequal contexts, and remains separate from generic CI.