AGENTS.md

AGENTS.md

Applies to the entire repository.

Purpose

tinyllm is a Rust-native LLM inference and serving framework. Reuse mature tensor/GPU primitives and focus original work on execution, scheduling, KV-cache management, memory ownership, serving, and observability. It is not a training stack, general tensor framework, or CUDA GEMM implementation.

Sources of truth

• README.md: commands, supported behavior, and limitations.
• docs/architecture.md: ownership boundaries and invariants.
• docs/roadmap.md: milestone scope and acceptance criteria.
• docs/dependencies.md: dependency policy.
• CONTRIBUTING.md: development conventions.
• benches/README.md: benchmark methodology.

Update the relevant document when behavior, architecture, dependencies, or commands change. Keep progress reports out of this file.

Architecture rules

• Keep artifact resolution, model semantics, runtime execution, and HTTP code separate.
• CLI and HTTP code are clients of the non-HTTP engine. They must not load weights, construct tensors, sample logits, or own KV caches.
• The synchronous model worker exclusively owns the tokenizer, model runner, device state, active sequences, logical cache allocator, and physical KV pool. Do not use Arc<Mutex<Model>> or execute the model on Tokio.
• Model code must not depend on request IDs, channels, CLI, or HTTP types. Engine-domain types must remain free of Candle and frontend types.
• Keep admission and channels bounded. Preserve ordered events, explicit overload, cooperative cancellation, and one terminal event per attached accepted request.
• Each uncached request starts with an empty block table. A prefix hit may start with immutable shared full pages representing exactly its computed prefix; suffix and decode writes remain private. Release all ownership, leases, and reservations on completion, cancellation, disconnect, failure, eviction, and shutdown.
• During chunked prefill, cache length and num_computed_tokens advance only after a synchronized chunk commit. Keep decode-first token-budget accounting and the single-prefill capacity guard intact.
• Prefer concrete Qwen3 implementations over speculative abstractions. Keep one Cargo package until an independent boundary justifies another crate.

Correctness and quality

• Generated prose is not correctness evidence. Validate tokenizer IDs, intermediate tensors/logits with declared tolerances, exact greedy token IDs, and cached-versus-uncached execution as applicable.
• Preserve num_computed_tokens <= total_sequence_tokens; cache length must equal the computed prefix, and terminal sequences must not execute again.
• Do not claim CUDA correctness from compilation or plausible output. Use a matching reference and identify the first divergent boundary.
• Keep unsafe_code = "forbid". Any future unsafe boundary requires documented invariants and focused tests.
• Use typed errors and structured parsers. Do not expose internal paths or backend details through public HTTP errors.
• Do not use unwrap, expect, todo!, dbg!, or placeholder panics in production code.
• Avoid unnecessary tensor copies, host/device transfers, cache reallocations, synchronizations, and per-token logging.
• Crossbeam remains the blocking engine channel; Tokio/Axum remain frontend dependencies.
• Follow the roadmap for large features. Do not introduce distributed inference, custom kernels, broad quantization, multimodal support, MoE, or speculative decoding early.
• Performance changes require a hypothesis, same-workload baseline, correctness evidence, and recorded environment metadata.

Validation

cargo fmt --all --check
cargo build --locked
cargo test --locked --all-targets
cargo clippy --locked --all-targets -- -D warnings

For model-path changes with the local checkpoint available:

cargo test --release --locked -- --ignored --test-threads=1

For CUDA-related changes:

cargo build --release --locked --features cuda