Support Qwen3.8-Flash-Next-GSQ-RCO GGUF and integrate Strata + KVMem #658

Before you start

• I have read the Roadmap and this is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.

Applies to

Desktop app

What problem does this solve

Hi FreeToken team,

I would like to request support for Qwen3.8-Flash-Next-GSQ-RCO GGUF. Many users are waiting to run this model locally, and FreeToken's edge-native MoE design looks like a great fit. Native GGUF support would make it much easier to load, test, and use this model without extra conversion work.

I also hope you can consider merging/integrating the Strata inference framework. Strata is built for MoE models that cannot fully fit in VRAM, splitting weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and could help run large models like Qwen3.8-Flash-Next on consumer hardware.

Please also consider integrating KVMem. Long-context agent workflows can quickly run out of VRAM, while KVMem keeps old KV history in host RAM and only loads relevant blocks into a bounded GPU working set. That would greatly improve long conversations, tool calls, and long-running sessions in FreeToken.

We really need these features. If any of them are already planned, please let us know. I am happy to help test or provide more details. Thank you!

Proposed solution

Add native support for Qwen3.8-Flash-Next-GSQ-RCO GGUF, so users can load and run this model directly in FreeToken without extra conversion work.

Integrate the Strata inference framework into FreeToken. Strata is designed for MoE models that cannot fully fit in VRAM. It splits weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and would help run large models like Qwen3.8-Flash-Next on consumer hardware.

Integrate KVMem into FreeToken as an optional KV cache / context manager. KVMem keeps long KV history in host RAM and only loads relevant blocks into a bounded GPU working set. This would improve long-context chat, tool calls, and long-running agent sessions without using too much VRAM.

Ideally, expose these features as config options so users can enable or disable them. Please also keep compatibility with FreeToken's current caching and memory management. I am happy to help test if needed.

Alternatives considered

No response