perf: add optional o200k SIMD masks

[clouds56]

Summary

• add o200k to the optional simd feature behind the existing NEON/AVX2 backends
• preserve the lighter GPT-2/r50k/cl100k classifier path with o200k-specific uppercase and slash masks
• add o200k rows to the automated SIMD benchmark report and document the local measurement

Implementation

Pattern::O200k now selects a dedicated pure-ASCII SIMD boundary scheme. It
models the o200k-only parts of the PAT expression:

• lowercase-to-uppercase word splits such as camelCase
• word-attached, case-insensitive contractions such as can't and x'll'd
• one-to-three digit groups
• CR/LF/slash punctuation tails and newline-aware whitespace

The scheme is deliberately conservative. It only activates for an ASCII first
64-byte window and only caches ends through byte 60. Unicode, short inputs, and
right-edge ambiguity remain on the existing scalar scanner. The public iterator
and the scalar result are unchanged.

O200k uses a separate NEON/AVX2 classification pass because its grammar needs
strict-uppercase and slash masks. GPT-2/r50k/cl100k keep their existing mask
set and therefore pay no extra classification cost.

Local benchmark

Apple M2 / Rust 1.97, 5 MiB fixture, nine paired samples:

Pattern / corpus	Scalar	NEON	Scalar → NEON
o200k English	14.44 ms	9.85 ms	1.47×
o200k Chinese control	8.42 ms	8.42 ms	1.00×

Both builds passed the direct fancy-regex correctness gate. The English
token-stream fingerprint was identical (1,233,584 pretokens); Chinese remains
a scalar control because its first window is non-ASCII.

The benchmark-comment renderer now includes o200k. Its trusted PR comment is
rendered from master, so this PR's bot comment will not list o200k until the
change lands; the benchmark job's candidate preview does.

Validation

• cargo test --all --features "cli simd" --verbose
• cargo test -p ffbpe-pat --features simd --verbose
• cargo check -p ffbpe-pat --features simd --target x86_64-apple-darwin
• node --test .github/scripts/update-benchmark-comment.test.cjs
• bash .github/scripts/validate-benchmark-reports.test.sh
• uv run --no-sync mkdocs build --strict
• targeted rustfmt --check for changed PAT sources
• scalar and --features simd 5 MiB, nine-sample PAT scan reports

[github-actions]

Benchmark report

✅ All base and PR correctness gates passed.

Compared b16fb70 → eb2b872 sequentially on the same runner. Timing deltas are informational.

Cells marked missing are absent cases or optional reports; unavailable means a required report is missing or a report is invalid; failed means that revision failed its report or case; changed means the workloads are not comparable.

Benchmark	Base	PR	Δ
Trainer — English byte, vocab 300 (exact)	8.87 ms	8.67 ms	-2.2%
Trainer — English byte, vocab 300 (k4096)	9.83 ms	9.58 ms	-2.6%
Trainer — English byte, vocab 1k (exact)	19.85 ms	19.85 ms	-0.0%
Trainer — English byte, vocab 1k (k4096)	21.23 ms	20.96 ms	-1.3%
Trainer — Chinese Unicode, vocab 300 (exact)	263.60 ms	245.34 ms	-6.9%
Trainer — Chinese Unicode, vocab 300 (k4096)	226.04 ms	217.47 ms	-3.8%
Trainer — Chinese Unicode, vocab 1k (exact)	445.64 ms	413.61 ms	-7.2%
Trainer — Chinese Unicode, vocab 1k (k4096)	397.33 ms	371.42 ms	-6.5%
Trainer — Chinese Unicode BBPE, vocab 1k (exact)	416.58 ms	403.77 ms	-3.1%
Trainer — Chinese Unicode BBPE, vocab 1k (k4096)	380.81 ms	354.89 ms	-6.8%
Pretokenizer — bigram pass	38.76 ms	38.61 ms	-0.4%
Pretokenizer — word pass	122.89 ms	120.32 ms	-2.1%
Pretokenizer — total	161.64 ms	158.93 ms	-1.7%
Codec — byte encode	50.62 ms	51.43 ms	+1.6%
Codec — byte decode	14.60 ms	14.69 ms	+0.6%
Codec — Unicode encode	141.87 ms	139.62 ms	-1.6%
Codec — Unicode decode	15.58 ms	15.54 ms	-0.3%
Codec — Unicode BBPE encode, vocab 1k	44.84 ms	45.69 ms	+1.9%
Codec — Unicode BBPE decode, vocab 1k	18.94 ms	18.73 ms	-1.1%

Pretokenizer pattern scan

Median scan time over paired samples (5.0 MiB per corpus); PR vs regex compares normal dispatch with direct fancy-regex on the PR revision.

Pattern / corpus	Base dispatch	PR dispatch	Δ	PR vs regex
GPT-2 — english	14.82 ms	14.94 ms	+0.8%	16.35×
GPT-2 — chinese	11.51 ms	11.62 ms	+1.0%	9.15×
r50k — english	14.82 ms	14.96 ms	+0.9%	21.76×
r50k — chinese	11.46 ms	11.59 ms	+1.2%	12.65×
cl100k — english	18.80 ms	18.80 ms	+0.0%	19.41×
cl100k — chinese	9.95 ms	9.96 ms	+0.1%	9.43×
o200k — english	20.17 ms	20.59 ms	+2.1%	9.46×
o200k — chinese	13.54 ms	13.55 ms	+0.1%	4.03×
unknown fallback — english	176.10 ms	170.27 ms	-3.3%	1.08×
unknown fallback — chinese	60.19 ms	59.22 ms	-1.6%	1.02×

Optional SIMD pattern scan

Revision	SIMD configuration
Base	simd enabled, AVX2 available
PR	simd enabled, AVX2 available

Median scan time over paired samples (5.0 MiB per corpus). The optional backend applies to ASCII-led GPT-2, r50k, and cl100k inputs; Chinese rows are scalar controls.

Pattern / corpus	Base simd	PR simd	Feature Δ	PR scalar → simd
GPT-2 — english	10.43 ms	10.21 ms	-2.2%	1.46×
GPT-2 — chinese (scalar control)	12.37 ms	11.44 ms	-7.6%	1.02×
r50k — english	10.46 ms	10.22 ms	-2.3%	1.46×
r50k — chinese (scalar control)	12.38 ms	11.40 ms	-7.9%	1.02×
cl100k — english	11.38 ms	11.23 ms	-1.4%	1.67×
cl100k — chinese (scalar control)	10.21 ms	9.89 ms	-3.1%	1.01×

Peak RSS

Benchmark	Base	PR	Δ
Trainer — English byte, vocab 300 (exact)	8.4 MiB	8.4 MiB	+0.5%
Trainer — English byte, vocab 300 (k4096)	8.5 MiB	8.5 MiB	+0.1%
Trainer — English byte, vocab 1k (exact)	11.3 MiB	11.4 MiB	+0.8%
Trainer — English byte, vocab 1k (k4096)	11.7 MiB	11.7 MiB	+0.4%
Trainer — Chinese Unicode, vocab 300 (exact)	110.8 MiB	109.5 MiB	-1.1%
Trainer — Chinese Unicode, vocab 300 (k4096)	84.8 MiB	84.8 MiB	+0.0%
Trainer — Chinese Unicode, vocab 1k (exact)	142.7 MiB	142.4 MiB	-0.2%
Trainer — Chinese Unicode, vocab 1k (k4096)	102.3 MiB	102.3 MiB	+0.0%
Trainer — Chinese Unicode BBPE, vocab 1k (exact)	142.0 MiB	142.0 MiB	+0.0%
Trainer — Chinese Unicode BBPE, vocab 1k (k4096)	99.8 MiB	99.8 MiB	+0.0%
Pretokenizer	44.9 MiB	44.8 MiB	-0.2%
Codec — byte encode	44.9 MiB	45.0 MiB	+0.0%
Codec — byte decode	44.9 MiB	45.0 MiB	+0.0%
Codec — Unicode encode	52.4 MiB	52.0 MiB	-0.9%
Codec — Unicode decode	44.9 MiB	45.0 MiB	+0.0%
Codec — Unicode BBPE encode, vocab 1k	44.9 MiB	45.0 MiB	+0.0%
Codec — Unicode BBPE decode, vocab 1k	44.9 MiB	45.0 MiB	+0.0%

Open benchmark run