Repository navigation
Conversation
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 11, 2026
…s on windows setup.py conflict with FlashML-org#575's Windows build: the row store takes next's shared extra_compile (/O2 on Windows, BuildExtension adds the /std flag; -O3 + the CUDA or ROCm C++ standard elsewhere) instead of the PR's own per-platform flags.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #384; complements #575.
Windows source builds omit
_row_store, and the implementation depends on POSIX file I/O. This prevents using disk-backed PLE tables on Windows.Implement the existing platform seams with Unicode file handles, overlapped positioned reads, aligned allocation and CUDA driver symbol loading. This lets fixed-width table rows remain on disk and enter the existing bounded staging buffers. The PLE hash and Linux reader algorithms retain their current behavior.
Extend the row-store tests with Unicode paths, offsets above 4 GiB, and recovery after a short read. Give the graph test's second store its own checkpoint file so a Windows writer does not overwrite a still-open file.
Validation: Windows 11 build 26200, RTX 5090 D v2 24 GB (driver 610.88), Ryzen 9 9900X, 64 GB RAM, MSVC 19.44, CUDA 13.0, Python 3.12.13, Torch 2.11.0+cu130. Base:
74a387777708b9650a027494bcb37d2ab9cf2443.11 passed, including GPU-oracle and CUDA-graph checks. Independently rebuilt against latest-main's Torch 2.14.1 CPU dependency: 8 passed, 3 GPU tests skipped (commands in the linked validation records). The baseline row-store source fails to compile on
dlfcn.hwith both versions.The harness (source and test logs) selects the declared row-store extension and builds the pinned-memory prerequisite with Windows compiler/library settings. Synthetic byte tables and toy PLE checkpoints were used. Full-model serving, Linux hardware and Torch 2.14 CUDA were not validated; the 2.11 CUDA run does not verify main's complete dependency contract.
Prepared with AI assistance. The contributor has reviewed and understands the change; the reported tests were executed through development tooling.