fix(tools): k3_tokenizer --ctest writes its fixture to a hardcoded /tmp path

[Unknown-Findout]

k3_tokenizer.py --ctest writes its case fixture to a hardcoded path:

cases = "/tmp/k3_tok_cases.bin"
with open(cases, "wb") as f:

This is the same pattern the repo has already moved away from twice, and I only found it because the existing pins pointed at it.

The rule is already written in this repo, twice

tests/test_stops.c:84 and tests/test_pipe_block.c:146:

Relative to the CWD, like test_compat_direct's TMPF - NOT "/tmp/...". These binaries are built by MinGW into native Windows .exe files, which resolve Windows paths: "/tmp" is not one, so mkdtemp there fails ENOENT and the whole make check goes red on the windows job (and only there).

And setup.sh was moved off its own /tmp probe to OMP_PROBE=".colibri-omp-probe-$$", with test_cuda_test_makefile.test_setup_openmp_probe_does_not_require_tmp pinning it there. tools/k3_tokenizer.py is the file that pass missed.

One correction to the obvious framing, because I checked

I expected this to fail outright on Windows and it did not. fopen(..., "w") creates the file but not the directory, and Git Bash creates C:/tmp on install, so on my box the old line writes fine. The failure is conditional on whether /tmp already exists, which is exactly why it survived this long: it works on most dev machines and breaks on clean ones. I measured it both ways rather than assuming, since assuming is what would have made this a bogus report.

The fixed filename is the platform-independent half, and it is unconditional: two concurrent --ctest runs collide on the same path, and nothing ever removed the file.

The change

tempfile.mkstemp next to the output file, in a try/finally:

fd, cases = tempfile.mkstemp(prefix=".colibri-k3-cases-", suffix=".bin",
                             dir=os.path.dirname(os.path.abspath(out)))

The output directory rather than the system temp dir on purpose: it is provably writable, because tokenizer.json was written to it successfully a few lines earlier. So this cannot trade one unwritable location for another.

Verification

Three tests in tests/test_k3_tokenizer_tmp.py, same shape as the existing setup.sh pin: no quoted /tmp path in the file, the fixture goes through mkstemp, and os.unlink sits inside the finally rather than on the happy path.

Negative control run rather than assumed - reverting tools/k3_tokenizer.py alone fails all three, restoring it passes all three:

revert tool -> FAILED (failures=3)
restore     -> OK

Full python suite, before and after:

Ran 591 tests   errors=2, skipped=64      (identical both ways)

Both errors are pre-existing and unrelated: test_cuda_test_makefile shells out to make, which is absent on this host.

The --ctest path itself is not exercised here, and I want to be straight about that - it needs tiktoken plus a compiled test_tok_kimi, so the tests read the source as text the way the setup.sh pin does. The edit is small enough to read, but CI is the check that matters for the runtime path.

Tested on Windows, RTX 3090 (sm_86), CUDA 13.1, Python 3.11.