docs: replace fabricated command output with real output, add a guard - #77
Merged
Conversation
Audited the command pages that had not already been corrected during #53-#62. Five of the six remaining were wrong: cmd-stats documented distinct_paths, document, total_nodes (real output is a single `paths` array) cmd-anomalies documented anomaly_count, records_analyzed, structural_deviations, and misnamed rarity_outliers / type_instability (real: rare_values, type_instabilities) cmd-query documented a {function, path, result} envelope (query emits the bare result) cmd-cascade documented chains, entities, records, response_events, trigger_events (real: cascades, total_cascades, total_events) cmd-essence documented a `vajra_essence` wrapper (real: identity, structure, anomalies, observations) Only cmd-inspect was accurate, though it was missing structural_findings from #61 — now documented, including that the field is omitted when empty, so absence means "nothing recognised" rather than "not checked". Across the session that is 10 of 11 command pages carrying invented schemas. A fabricated schema is worse than no docs: a `jq` filter written against `distribution_shifts` returns silence rather than an error. Each block is now real captured output, with a short note on how to read it rather than more example. Where a real value is unhelpfully long (a 64-char hash, a 20-entry array) it is trimmed, never invented. Adds vajra-cli/tests/docs_examples.rs, which asserts that the top-level keys in each page's first JSON example are keys the command actually emits. Subset rather than equality, since a page may reasonably trim an example and some fields are omitted when empty — what must never happen is a documented key the command cannot produce. Verified the guard fails: injecting a fake `records_analyzed` into cmd-stats.md produces cmd-stats.md documents keys ["stats"] never emits: ["records_analyzed"] Closes #73.
The workspace denies clippy::panic, and CI's `cargo clippy --workspace` does not lint test targets, so this would have passed CI while violating the guideline.
📝 WalkthroughWalkthroughThe PR updates six command-page JSON examples and adds Rust integration tests that compare documented top-level keys with keys emitted by the CLI using deterministic fixtures. ChangesCLI JSON documentation synchronization
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/src/cmd-essence.md`:
- Line 164: Update the documentation around the ranking output description to
remove the obsolete meta.truncated guidance and any claim that relies on the
removed meta field. Ensure the text reflects the current schema, which exposes
scoring components and score without a meta object.
In `@docs/src/cmd-query.md`:
- Around line 146-149: Update the vajra query invocation in the
contributor-diversity threshold example to request JSON output before piping its
numeric result to bc, while preserving the existing entropy expression and
comparison logic.
In `@vajra-cli/tests/docs_examples.rs`:
- Line 141: Update the command-status assertion in the docs example test to
return an anyhow error containing the command’s stderr when out.status is
unsuccessful, instead of discarding the diagnostic output. Preserve the existing
success path and follow the contextual error-handling pattern used by real_keys.
- Around line 91-95: Scope optional-key exemptions to each individual command
case instead of using the global optional BTreeSet in the documentation test.
Update the case data and validation logic around the docs example fixtures so
each command explicitly declares only the keys it may omit, while preserving the
existing behavior for legitimately optional keys.
- Around line 37-52: Extend the documentation test around documented_keys and
the query case to validate the JSON shape, not only object keys: add
cmd-query.md to the cases covered and assert that its documented JSON block
parses as a scalar value, then run the real query with --format json and assert
its output is also scalar. Keep existing object-key checks unchanged for other
command examples.
- Around line 20-27: Update the vajra_bin function to obtain the executable path
from the CARGO_BIN_EXE_vajra environment variable instead of manually
constructing target/debug/vajra. Preserve the function’s PathBuf return type and
allow Cargo to provide the correct profile, target directory, and
platform-specific executable name.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 441241e8-a069-4122-b5cf-e18fd5b250ea
📒 Files selected for processing (7)
docs/src/cmd-anomalies.mddocs/src/cmd-cascade.mddocs/src/cmd-essence.mddocs/src/cmd-inspect.mddocs/src/cmd-query.mddocs/src/cmd-stats.mdvajra-cli/tests/docs_examples.rs
| } | ||
| ``` | ||
|
|
||
| Each entry carries its scoring components alongside the final `score`, so a ranking is decomposable — `--explain` prints the same weights in text form. The active profile's weight vector determines how the components combine. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Remove the obsolete meta.truncated guidance.
The updated output no longer contains meta, but the following paragraph still directs users to meta.truncated. Update or remove that truncation claim to match the new schema.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/src/cmd-essence.md` at line 164, Update the documentation around the
ranking output description to remove the obsolete meta.truncated guidance and
any claim that relies on the removed meta field. Ensure the text reflects the
current schema, which exposes scoring components and score without a meta
object.
Comment on lines
+146
to
+149
| ```bash | ||
| if (( $(echo "$(vajra query data.json 'entropy($[*].author)') < 3" | bc -l) )); then | ||
| echo "contributor diversity below threshold" | ||
| fi |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Request JSON output before passing the result to bc.
Line 147 uses the default text format, so bc receives the human-readable report instead of a numeric scalar.
Proposed fix
-if (( $(echo "$(vajra query data.json 'entropy($[*].author)') < 3" | bc -l) )); then
+if (( $(echo "$(vajra query data.json 'entropy($[*].author)' --format json) < 3" | bc -l) )); then📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ```bash | |
| if (( $(echo "$(vajra query data.json 'entropy($[*].author)') < 3" | bc -l) )); then | |
| echo "contributor diversity below threshold" | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/src/cmd-query.md` around lines 146 - 149, Update the vajra query
invocation in the contributor-diversity threshold example to request JSON output
before piping its numeric result to bc, while preserving the existing entropy
expression and comparison logic.
Comment on lines
+20
to
+27
| fn vajra_bin() -> PathBuf { | ||
| let mut p = PathBuf::from(env!("CARGO_MANIFEST_DIR")); | ||
| p.pop(); | ||
| p.push("target"); | ||
| p.push("debug"); | ||
| p.push("vajra"); | ||
| p | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)vajra-cli/(tests/docs_examples\.rs|Cargo\.toml|Cargo\.lock)$' || true
echo "== docs_examples.rs =="
if [ -f vajra-cli/tests/docs_examples.rs ]; then
cat -n vajra-cli/tests/docs_examples.rs | sed -n '1,80p'
fi
echo "==vajra-cli Cargo.toml =="
cat -n vajra-cli/Cargo.toml | sed -n '1,220p'
echo "== references to vajra_bin / CARGO_BIN_EXE =="
rg -n 'vajra_bin|CARGO_BIN_EXE|target/debug/vajra' .Repository: copyleftdev/vajra
Length of output: 11646
Use Cargo’s integration-test binary path.
vajra-cli/tests/docs_examples.rs currently constructs target/debug/vajra, which can break under custom CARGO_TARGET_DIR, non-default profiles, or platform executable suffixes. Use CARGO_BIN_EXE_vajra in tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@vajra-cli/tests/docs_examples.rs` around lines 20 - 27, Update the vajra_bin
function to obtain the executable path from the CARGO_BIN_EXE_vajra environment
variable instead of manually constructing target/debug/vajra. Preserve the
function’s PathBuf return type and allow Cargo to provide the correct profile,
target directory, and platform-specific executable name.
Comment on lines
+37
to
+52
| /// Top-level keys of the first parseable JSON object in a page. | ||
| fn documented_keys(page: &Path) -> Result<Option<BTreeSet<String>>> { | ||
| let text = | ||
| std::fs::read_to_string(page).with_context(|| format!("cannot read {}", page.display()))?; | ||
| let mut rest = text.as_str(); | ||
| while let Some(start) = rest.find("```json") { | ||
| let after = &rest[start + 7..]; | ||
| let Some(end) = after.find("```") else { break }; | ||
| let body = &after[..end]; | ||
| if let Ok(serde_json::Value::Object(map)) = serde_json::from_str::<serde_json::Value>(body) | ||
| { | ||
| return Ok(Some(map.keys().cloned().collect())); | ||
| } | ||
| rest = &after[end..]; | ||
| } | ||
| Ok(None) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate query’s scalar JSON contract too.
documented_keys ignores scalar JSON and cmd-query.md is absent from cases, so an envelope regression—or a conflicting query example—passes this guard. Add a query-specific assertion that both the documented block and real --format json output are scalar values.
Also applies to: 97-102
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@vajra-cli/tests/docs_examples.rs` around lines 37 - 52, Extend the
documentation test around documented_keys and the query case to validate the
JSON shape, not only object keys: add cmd-query.md to the cases covered and
assert that its documented JSON block parses as a scalar value, then run the
real query with --format json and assert its output is also scalar. Keep
existing object-key checks unchanged for other command examples.
Comment on lines
+91
to
+95
| // Optional keys the command omits when empty, so a page may document them | ||
| // even though this fixture does not trigger them. | ||
| let optional: BTreeSet<&str> = ["structural_findings", "windows", "trends"] | ||
| .into_iter() | ||
| .collect(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Scope optional-key exemptions per command.
This global allowlist lets any tested page document structural_findings, windows, or trends without failure, even when that command never emits the key. Associate optional keys with each individual case.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@vajra-cli/tests/docs_examples.rs` around lines 91 - 95, Scope optional-key
exemptions to each individual command case instead of using the global optional
BTreeSet in the documentation test. Update the case data and validation logic
around the docs example fixtures so each command explicitly declares only the
keys it may omit, while preserving the existing behavior for legitimately
optional keys.
| .args(["--response-values", "fix"]) | ||
| .args(["--format", "json", "--quiet"]) | ||
| .output()?; | ||
| assert!(out.status.success()); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Return cascade failures with stderr context.
Unlike real_keys, this assertion discards the command’s stderr, making fixture or argument failures needlessly opaque. Return an anyhow! error including stderr.
As per coding guidelines, “All errors must include enough context to diagnose the failure without a debugger.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@vajra-cli/tests/docs_examples.rs` at line 141, Update the command-status
assertion in the docs example test to return an anyhow error containing the
command’s stderr when out.status is unsuccessful, instead of discarding the
diagnostic output. Preserve the existing success path and follow the contextual
error-handling pattern used by real_keys.
Source: Coding guidelines
copyleftdev
added a commit
that referenced
this pull request
Jul 30, 2026
…ollow-ups Two CodeRabbit findings on #77 that I merged before reading, plus the user-visible half of #75. The format gap, measured: `--format markdown` and `--format compact-ai` produce byte-identical output to `text` for eleven of the twelve commands. Only `essence` has a real renderer. The CLI advertises four formats and delivers two. Accepting a format flag and ignoring it is the same failure as reporting `errors: []` over a partial batch — the caller cannot distinguish "rendered as Markdown" from "fell back to text". Those commands now say so on stderr, name a format that does work, stay silent under --quiet so pipelines are unaffected, and leave stdout byte-identical. This does not close the renderer gap; it stops the interface lying about it. Docs follow-ups from the #77 review: - cmd-essence.md referenced `meta.truncated` next to the JSON example I had replaced. CodeRabbit read this as obsolete guidance for a removed key. Checking the actual output, `meta` is real — it belongs to the `--format compact-ai` shape (meta, doc, notable, motifs, drill, v), not to `--format json`. So the fix is to say which format carries it rather than delete accurate documentation. - cmd-query.md's shell example now passes --format json --quiet explicitly. The example was not broken — text mode already emits a bare number — but depending on that implicitly is the kind of assumption that stops being true without warning. Tests: 5 covering the format notice (announced for both unimplemented formats and points at json, silent for essence, silent for text/json, suppressed by --quiet, stdout unchanged). Refs #75.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #73.
What was wrong
Audited the command pages not already corrected during #53–#62. Five of the six remaining were wrong:
cmd-statsdistinct_paths,document,total_nodes,pathspathscmd-anomaliesanomaly_count,records_analyzed,structural_deviations,rarity_outliers,type_instabilitynumeric_outliers,rare_values,type_instabilitiescmd-query{function, path, result}envelope1.584962500721156)cmd-cascadechains,entities,records,response_events,trigger_eventscascades,total_cascades,total_eventscmd-essencevajra_essencewrapper objectidentity,structure,anomalies,observationsOnly
cmd-inspectwas accurate — though it was missingstructural_findingsfrom #61, now documented including the fact that the field is omitted when empty, so its absence means "nothing recognised" rather than "not checked".Session total: 10 of 11 command pages carried invented schemas. That's the part worth internalising — it wasn't a stale page here and there, it was the default state.
Why it matters more than it looks
A fabricated schema is worse than no docs.
jq '.distribution_shifts'against realdriftoutput doesn't error — it returnsnull. Someone builds a CI gate on that and it silently never fires.What changed
Each JSON block is now real captured output, plus a short note on how to read it rather than more example. Where a real value is unhelpfully long (a 64-char hash, a 20-entry array) it's trimmed — never invented. Some of the notes are the useful part:
cmd-anomalies:z_madis deviation from the median in MAD units, so it's robust to the outliers it's detecting.cmd-query: the bare-result design is what makes it composable, with a shell example.cmd-cascade:self_fix_rateis the share where trigger and response share an author — a low rate means someone else is cleaning up.cmd-essence: entries carry their scoring components alongside the final score, so a ranking is decomposable.The guard
vajra-cli/tests/docs_examples.rsasserts that the top-level keys in each page's first JSON example are keys the command actually emits.Subset rather than equality, deliberately: a page may reasonably trim an example, and some fields are omitted when empty (
structural_findings,windows,trendsare allow-listed). What must never happen is a documented key the command cannot produce.Verified the guard actually fails. Injecting a fake key into
cmd-stats.md:and passing again once reverted. A guard that can't fail isn't a guard.
Note
The second commit fixes a
clippy::panicviolation I introduced in the guard itself. The workspace deniesclippy::panic, but CI'scargo clippy --workspacedoesn't lint test targets — so it would have passed CI while breaking the guideline. That's #75's first half in miniature.Full workspace suite: 78 test binaries, 0 failures.
cargo fmt --checkclean. 0 clippy errors invajra-cliunder--all-targets --all-features.Summary by CodeRabbit
Documentation
Tests