Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ Vajra is a deterministic semantic reduction and anomaly analysis engine for JSON

## The Rules

1. **Every algorithm must work at any scale.** 1KB, 1GB, 100GB. If it doesn't stream, it doesn't ship.
1. **Every algorithm must work at any scale.** 1KB, 1GB, 100GB. Design accumulators to bounded memory: sketches over exact collections, one pass where one pass suffices, a documented accuracy bound where it does not.

This is the target, and the codebase does not meet it yet — there is no incremental parser, so every analysis materialises the corpus first (#102). Stating the rule as "if it doesn't stream, it doesn't ship" made it unfalsifiable: two commands were flagged for violating it while nothing in the tool actually streamed. A new analysis should say what its memory scales with, and `separation`'s docs are the example to follow.
2. **Determinism is sacred.** Same input + same config = same output. Always. Use `BTreeMap` for any externally-visible ordering. Seed all randomized algorithms. Fixed traversal order everywhere.
3. **Strong primitives only.** No speculative ML, no O(n^2) algorithms, no techniques requiring tuning. Published, peer-reviewed, deployed at scale — or it doesn't belong.
4. **Honest inference.** Every heuristic is labeled. Every score is decomposable. Never silently assert a guess as truth.
Expand Down Expand Up @@ -216,7 +218,7 @@ Agent(isolation: worktree) → vajra-anomaly
- No `unsafe` blocks.
- No `HashMap` where output order matters. Use `BTreeMap`.
- No `f64` equality. Use relative tolerance.
- No `println!` for output. All user-facing output through the rendering system.
- No `println!` for building output. Text and Markdown are built as a `render::Report` and written once; `println!` appears at most once per format branch, to emit the finished string. Building output line by line duplicates format handling per command, which is how `--format markdown` silently fell through to text in several of them.
- No `#[allow(clippy::...)]` without a comment explaining why the lint is wrong here.
- No dependencies without checking: maintained? reasonable dep tree? `no_std` compatible where needed?
- No `unwrap()`, `expect()`, or `panic!()`. Ever. The clippy deny lints enforce this.
Expand Down
25 changes: 22 additions & 3 deletions docs/src/streaming.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,25 @@
# Streaming

Vajra handles JSON of any size. A 50 KB medical claim and a 10 GB event log enter the same pipeline. The streaming engine is what makes this possible.
> **Status: this page describes the intended design, not the current implementation.**
>
> There is no SAX parser yet. `load_adaptive`'s large-input branch reads the
> whole file, parses a full DOM, and then materialises a `Vec<JsonEvent>`
> alongside it — so `--streaming` currently uses *more* memory than the default
> path, not less. Measured on a 15 MB, 120,000-record input:
>
> | mode | peak RSS |
> |---|---|
> | default (DOM) | 233 MB |
> | `--streaming` | 402 MB |
>
> The flag selects the sketch-based accumulators, which are real and tested;
> what is missing is an incremental parser to feed them. Passing `--streaming`
> prints a warning saying so. Tracked in
> [#102](https://github.com/copyleftdev/vajra/issues/102). The memory budget
> below is the target the accumulators were designed against, and holds once a
> record iterator exists that never materialises the corpus.

Vajra aims to handle JSON of any size: a 50 KB medical claim and a 10 GB event log entering the same pipeline. The streaming engine is what would make this possible.

---

Expand All @@ -16,7 +35,7 @@ For documents that fit in memory. The parser builds a full in-memory tree with r

### Streaming Mode

For documents that exceed available memory. SAX-style event parsing with bounded memory. The parser emits events (start-object, key, value, end-object, start-array, end-array) and the analyzers update their accumulators incrementally.
*Intended:* for documents that exceed available memory, SAX-style event parsing with bounded memory. *Today:* the events are produced by walking a fully parsed DOM, so this mode costs more memory than the DOM mode it is meant to replace (#102). The parser emits events (start-object, key, value, end-object, start-array, end-array) and the analyzers update their accumulators incrementally.

**Memory:** O(p + s) where p = distinct paths and s = sum of sketch sizes. For typical JSON with < 1,000 distinct paths: < 10 MB regardless of document size.
**Activates:** Automatically when document size exceeds the streaming threshold. Force with `--streaming`.
Expand Down Expand Up @@ -119,7 +138,7 @@ Fingerprint: ~10 KB
Total: ~5.2 MB
```

This budget holds regardless of whether the document is 100 MB or 100 GB. The streaming guarantee: **bounded memory independent of input size**.
This budget is what the accumulators were designed to hold to, regardless of whether the document is 100 MB or 100 GB. It is not yet achieved: the accumulators are fed from a fully materialised event vector, so today's memory scales with input size. See the status note at the top of this page and [#102](https://github.com/copyleftdev/vajra/issues/102).

---

Expand Down
Loading
Loading