docs: show the two commands working together - #172
Conversation
Neither README.md nor docs/cli.md said how classifier and keywords relate. Each was documented alone, so a reader met two tools and no reason to run them together. Add three workflows, each verified against the installed 2.7.0 binaries: Read the label and the reason side by side. classifier -p gives the verdict and keywords -n gives the terms that carried it, which is what you want when a classification surprises you. This is the pattern the explainability literature calls showing top terms alongside the predicted label. Find corpus-specific stopwords. A review corpus repeats "delivery" and no general stopword list knows it. --max-df drops a term above a document ratio, so comparing keywords info before and after names the term, and dropping it sharpens every term that carries signal. Keep the models straight. The two commands write different formats and reject each other, classifier takes -f, and keywords takes -m. Also warn against piping keywords into classifier. TF-IDF ranks a term by how well it separates one document from the corpus, not by how well it signals a category, so the top terms are not the strongest evidence. Measured on a long review, the full text classifies at 0.92 and its top four terms at 0.79.
Greptile SummaryThe PR documents how to use
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Reviews (2): Last reviewed commit: "docs: stop calling TF-IDF terms an expla..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| The first line is the verdict. The second says which terms drove it, which is | ||
| what you need when a classification surprises you. |
There was a problem hiding this comment.
When users follow this workflow, it presents corpus-wide TF-IDF terms from an independent model as the terms that drove the classifier verdict, causing users to attribute predictions to features that do not determine the category and make incorrect model-tuning decisions.
Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/cli.md
Line: 172-173
Comment:
**TF-IDF terms do not explain predictions**
When users follow this workflow, it presents corpus-wide TF-IDF terms from an independent model as the terms that drove the classifier verdict, causing users to attribute predictions to features that do not determine the category and make incorrect model-tuning decisions.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Valid, and fixed in a585814. You are right that this was the core claim of the section and it was wrong.
The two commands hold separate models. keywords reports TF-IDF weight, which measures how well a term separates one document from the rest of its corpus. It never sees the classifier and does not know which category a term favors, so a term can top that list and carry no weight in the decision. Calling it "which terms drove it" invites exactly the mis-tuning you describe.
Corrected in four places, in docs/cli.md and README.md. The section now says the terms are the document in shorthand, useful for checking that it says what you assumed, and adds an explicit warning:
The second line is not an explanation of the first. The two commands hold separate models.
keywordsreports TF-IDF weight, which measures how well a term separates this document from the rest of the corpus. It never sees the classifier, and it does not know which category a term favors. A term can top the list and carry no weight in the decision.
It now points at Classifier::LogisticRegression#weights for the weights a model actually holds, and notes that no CLI flag reports per-term weights for a Bayes model, which I confirmed against the source.
Worth noting the PR already made the same argument in its "Do not pipe one into the other" section, with measurements showing confidence dropping from 0.92 to 0.79. I made the opposite claim two sections earlier without noticing the contradiction. Good catch.
The side-by-side section said the keywords output showed "which terms drove" the classifier verdict. It does not. The two commands hold separate models. keywords reports TF-IDF weight, which measures how well a term separates one document from its corpus, and it never sees the classifier or learns which category a term favors. A term can top that list and carry no weight in the decision. Presenting it as attribution invites the reader to tune the wrong thing. Say what the terms are: the document in shorthand, useful for checking that it says what you assumed. Point at Classifier::LogisticRegression#weights for the weights a model actually holds, and note that no command line flag reports per-term weights for a Bayes model. Found by Greptile on #172.
1 participant
Neither
README.mdnordocs/cli.mdsaid howclassifierandkeywordsrelate. Each was documented alone, so a reader met two commands and no reason to run them together.Adds three workflows to
docs/cli.md, plus a short pointer in the README. Every console block was run against the installed 2.7.0 binaries and the output pasted from the real run.1. Read the label and the reason side by side
The verdict, then the terms that carried it. This is what the explainability literature calls showing top terms alongside the predicted label, and it is the thing you want when a classification surprises you.
2. Find corpus-specific stopwords
A term in nearly every document tells a classifier nothing, and every corpus grows its own. No general stopword list knows that a review corpus repeats
delivery.One term went. Scoring under each model names it:
deliverysat in all 12 documents and still drew weight. Dropping it sharpens every term that carries real signal. The section also warns that these bounds cut fast:--min-df 2took the same vocabulary from 42 terms to 7.3. Keep the models straight
The commands write different formats and reject each other, which is worth stating because the flags differ too (
-fvs-m):What I deliberately did not recommend
Piping
keywordsintoclassifier. It looks clever and it makes results worse:Confidence drops from 0.92 to 0.79. TF-IDF ranks a term by how well it separates one document from the corpus, not by how well it signals a category, so it puts
arrivedfirst, a neutral delivery word. The docs now say to classify the full text and usekeywordsto explain it.The literature does back TF-IDF feature selection feeding Naive Bayes, but as an in-pipeline weighting step, not as two CLIs piped together. The
--min-df/--max-dfworkflow above is the part of that idea these commands actually expose.Verified