hf-tokenize

Count and visualize the tokens of a file, with any tokenizer on the Hub.

An extension for the hf CLI. The hf CLI manages repos, files and caches, but it never tells you how a model will actually cut your text up. This adds that.

$ hf tokenize -m Qwen/Qwen3-0.6B README.md
$ hf tokenize -m Qwen/Qwen3-0.6B -c src/*.py
$ cat prompt.txt | hf tokenize -m meta-llama/Llama-3.1-8B-Instruct

Every count belongs to one tokenizer, so -m is required. There is no default model: a number that silently came from a tokenizer you did not choose is worse than no number. Name the one you usually want once, and drop the flag:

$ export HF_TOKENIZE_MODEL=Qwen/Qwen3-0.6B
$ hf tokenize README.md

Forget it and the error tells you what you already have:

$ hf tokenize README.md
Error: No model given. Pass -m, or set HF_TOKENIZE_MODEL.
Tokenizers in your cache:
  Qwen/Qwen2.5-1.5B-Instruct
  Qwen/Qwen3-0.6B
Any repo with a tokenizer.json works. Browse them at
  https://huggingface.co/models?pipeline_tag=text-generation&sort=trending

On a terminal every token gets its own background color, so you can see where the boundaries fall. Whitespace stays visible: tabs show as → and newlines as ⏎. Text that the tokenizer throws away is dimmed, and the summary line counts it.

Through a pipe the output turns into one token per line, ready for grep and cut:

$ hf tokenize -i src/main.py | head -3
0	1782	def
1	4402	 main
2	2258	():

Looking at less, and getting text back

--truncate is a limit, not a mode. It caps how many tokens are looked at and leaves the output alone, so the head of a big file stays readable:

$ hf tokenize -t 12 README.md
# hf-tokenize ...                       (colored, twelve tokens of it)
12 of 953 tokens · 50 chars · 4.2 chars/token · Qwen/Qwen3-0.6B

It applies to counting too: hf tokenize -c -t 12 README.md reports 12.

--text prints the text instead of the tokens. Together the two trim a prompt to fit a context window:

$ hf tokenize -t 4000 --text transcript.txt > prompt.txt
4,000 of 18,412 tokens · 16,203 chars · Qwen/Qwen3-0.6B

The cut lands on a token boundary and keeps the original characters byte for byte. It slices the text you gave it rather than decoding the tokens back, so nothing is normalized on the way out. The summary goes to standard error, so it never lands in the file.

Install

hf extensions install samlaf/hf-tokenize

The hf CLI installs it into its own virtual environment, so nothing lands in yours. Then hf tokenize works like any built-in command.

Options

Option	What it does
-m, --model MODEL	Hub repo id, a local directory, or a tokenizer.json. Required, or set HF_TOKENIZE_MODEL.
-c, --count	Print the token count only, like wc. Totals several files.
-i, --ids	Show the id of every token.
-t, --truncate N	Look at the first N tokens only. Applies to every mode.
--text	Print the text itself, not the tokens.
-s, --special	Add the special tokens of the model.
--color auto|always|never	Color the tokens. auto follows the terminal and NO_COLOR.

A whole model family almost always shares one tokenizer, so any member will do: every Qwen3 repo gives the same ids as Qwen/Qwen3-0.6B, and every Llama-3.x repo agrees with any other. Reach for the smallest one, and for a gated family reach for an ungated sibling.

The tokenizer comes from your Hub cache when it is already there, so a model you have downloaded works offline. Models that ship a slow (SentencePiece) tokenizer only have no tokenizer.json; the error message tells you how to convert one.

Development

uv run hf-tokenize README.md
uv run pytest

License

Apache-2.0