diff --git a/README.md b/README.md index c2a93f5..3cc2365 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,23 @@ space, as in `machine learning:0.35`. Run `keywords --help` for the full option list. A usage error exits 2 and any other error exits 1, so scripts can tell the two apart. -[keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) +Run the two commands side by side to read a label together with the terms that +make the text distinctive: + +```bash +classifier -f reviews-model.json -p "Broken on arrival, awful quality" +# => positive:0.12 negative:0.88 + +keywords -m reviews.json -n 5 "Broken on arrival, awful quality" +# => awful:0.52 arrival:0.52 broken:0.52 quality:0.44 +``` + +They keep separate models in separate formats, so `classifier` takes `-f` and +`keywords` takes `-m`, and neither reads the other's file. The terms are +context, not an explanation of the label: TF-IDF measures how well a term +separates a document from its corpus, not how much it favors a category. + +[Using both commands →](docs/cli.md#using-both-commands-together) · [keywords reference →](docs/keywords.md) · [CLI Guide →](https://rubyclassifier.com/docs/guides/cli/basics) ### Claude Code Plugin diff --git a/docs/cli.md b/docs/cli.md index afda9f5..2c38abe 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -135,6 +135,157 @@ $ classifier -m lsi related article.txt | `-v`, `--version` | Print the gem version | | `-h`, `--help` | Print the full usage | +## Using both commands together + +`classifier` and `keywords` answer different questions about the same text. +`classifier` gives a category. `keywords` names the terms that make the text +distinctive against its corpus. Run them side by side to read a label together +with what the document is about. + +Fit a vocabulary from the same corpus you train on, and the two views line up: + +```console +$ keywords fit -m reviews.json reviews/good/*.txt reviews/bad/*.txt +$ keywords info -m reviews.json +Documents: 8 +Vocabulary: 37 +Min DF: 1 +Max DF: 1.0 +``` + +`keywords info` reports the corpus before you commit to training. A vocabulary +of 37 terms over 8 documents says the corpus is far too small, and no +classifier fixes that. + +Then train, and read the two answers together: + +```console +$ classifier -f reviews-model.json train positive reviews/good/*.txt +$ classifier -f reviews-model.json train negative reviews/bad/*.txt + +$ classifier -f reviews-model.json -p "Broken on arrival, awful quality and useless customer service" +positive:0.07 negative:0.93 + +$ keywords -m reviews.json -n 5 "Broken on arrival, awful quality and useless customer service" +useless:0.4 awful:0.4 arrival:0.4 broken:0.4 service:0.34 +``` + +The first line is the verdict. The second is the document in shorthand, which +tells you what the classifier was reading when a result surprises you. + +**The second line is not an explanation of the first.** The two commands hold +separate models. `keywords` reports TF-IDF weight, which measures how well a +term separates this document from the rest of the corpus. It never sees the +classifier, and it does not know which category a term favors. A term can top +the list and carry no weight in the decision. + +Read it as context, not as attribution. When a label looks wrong, the terms +tell you whether the document says what you assumed, which is usually the real +problem. For the weights a model actually holds, use +`Classifier::LogisticRegression#weights` from Ruby, which returns the learned +weight per term and per category: + +```ruby +classifier.weights("positive", limit: 5) +``` + +No command line flag reports per-term weights for a Bayes model. + +### Find the words your corpus wastes on itself + +A term that appears in nearly every document tells a classifier nothing, and +every corpus grows its own. A review corpus repeats `delivery`, a support +corpus repeats `ticket`. These are stopwords that no general stopword list +knows about, because they are specific to your data. + +`keywords` finds them, because `--max-df` drops a term that appears in more +than the given ratio of documents. Compare the vocabulary size before and +after: + +```console +$ keywords fit -m default.json good.txt bad.txt +$ keywords info -m default.json +Documents: 12 +Vocabulary: 42 + +$ keywords fit -m pruned.json --max-df 0.5 good.txt bad.txt +$ keywords info -m pruned.json +Documents: 12 +Vocabulary: 41 +``` + +One term went. Score a document under each model to see which: + +```console +$ keywords -m default.json -n 3 "delivery was awful and broken" +broken:0.69 awful:0.69 delivery:0.24 + +$ keywords -m pruned.json -n 3 "delivery was awful and broken" +broken:0.71 awful:0.71 +``` + +`delivery` sat in all 12 documents and still drew weight. Dropping it sharpens +every term that carries real signal. + +Watch the vocabulary count as you tune, because these bounds cut fast: + +```console +$ keywords fit -m tight.json --min-df 2 good.txt bad.txt +$ keywords info -m tight.json +Documents: 12 +Vocabulary: 7 +``` + +`--min-df 2` took 42 terms down to 7. That is no longer a vocabulary, it is a +handful of words. Move one bound at a time and read `keywords info` after each +change. + +### Two commands, two models + +The models are separate files in separate formats, and neither command reads +the other's: + +```console +$ classifier -f reviews.json "broken awful" +Error: Unknown classifier type in model: tfidf + +$ keywords -m reviews-model.json "broken awful" +Error: Invalid vectorizer type: bayes +``` + +Note the flags differ too. `classifier` takes `-f`, and `keywords` takes `-m`. + +| | `classifier` | `keywords` | +|:--|:--|:--| +| Answers | Which category | Which terms matter | +| Model flag | `-f` | `-m` | +| Default model | `./classifier.json` | `./keywords.json` | +| Builds a model with | `train` | `fit` | +| Pre-trained models | Yes, through `-r` | No | + +### Do not pipe one into the other + +`keywords` prints `term:score` pairs, which is not text to classify. Feeding +its output to `classifier` throws away the rest of the document and weakens the +result: + +```console +$ classifier -f reviews-model.json -p "$LONG_REVIEW" +positive:0.08 negative:0.92 # the whole review + +$ keywords -m reviews.json -n 4 "$LONG_REVIEW" +arrived:0.6 refund:0.3 useless:0.3 build:0.3 + +$ classifier -f reviews-model.json -p "arrived refund useless build" +positive:0.21 negative:0.79 # weaker, from the top terms alone +``` + +Confidence drops from 0.92 to 0.79. TF-IDF ranks a term by how much it +distinguishes one document from the rest of the corpus, which is not the same +as how much it signals a category. Here it puts `arrived` first, a neutral +word about delivery. Classify the full text, and read `keywords` alongside it +for context rather than for attribution. + ## Install without Ruby Homebrew installs the command line tools on their own: