Repository navigation
fix(server): allow overriding the distributed port #301
Description
Summary
ServerArgs.distributed_addr (python/freetoken/server/args.py) is fixed
to tcp://127.0.0.1:{self.server_port + 1}. If anything else listens on
that neighbour port, the backend worker dies at startup with
DistNetworkError: … EADDRINUSE and the API server shuts down. There is
no flag or environment variable to move it.
On our machine, --port 8081 collided with a Docker proxy listening on
127.0.0.1:8082.
Environment
- FreeToken 0.1.2, source install; upstream base
4b94bdc. (Our checkout carries an opt-in localFREETOKEN_DIST_PORToverride — the env var was NOT set for this run, so the reproduced behaviour is the upstream default.) torch 2.11.0+cu130, CUDA toolkit 13.0.2, driver 595.84 - RTX 3060 Laptop 6 GB (sm_86), Ryzen 9 5900HX, 30 GB RAM, Linux Mint 22.3
- Checkpoint:
nvidia/Qwen3.6-35B-A3B-NVFP4
Reproduce
With any listener on server_port + 1 (e.g. python3 -m http.server 8082 --bind 127.0.0.1):
ft serve --model /home/user/models/Qwen3.6-35B-A3B-NVFP4 --port 8081Key lines:
ERROR Backend supervisor: DistNetworkError: The server socket has failed to listen
on any local network address. port: 8082, useIpv6: false, code: -98,
name: EADDRINUSE, message: address already in use
ERROR Backend worker is gone and cannot be restarted; stopping the API server
Full log
[2026-08-31|00:36:44] INFO Parsed arguments:
ServerArgs(model_path='/home/user/models/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=3049384', server_host='127.0.0.1', server_port=8081, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.6-35B-A3B-NVFP4', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', thinking='auto', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=(), gpu_assigned=None)
[2026-08-31|00:36:44|FrontendAPI] INFO Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO: Started server process [3049384]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8081 (Press CTRL+C to quit)
INFO: 127.0.0.1:56362 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56366 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56396 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56412 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56446 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56452 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56474 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56480 - "GET /metrics HTTP/1.1" 404 Not Found
/home/user/dev/local-ai/upstream/FreeToken/python/freetoken/engine/engine.py:1030: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-08-31|00:36:47|core|rank=0] INFO Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
INFO: 127.0.0.1:56506 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56512 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56540 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56546 - "GET /metrics HTTP/1.1" 404 Not Found
[2026-08-31|00:36:47|core|rank=0] INFO Auto-selected attention backend: fi
[2026-08-31|00:36:47|core|rank=0] INFO benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[2026-08-31|00:36:47|core|rank=0] INFO Auto-selected MoE backend: hybrid
[2026-08-31|00:36:47|core|rank=0] INFO No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[2026-08-31|00:36:47|core|rank=0] INFO Resolved config: moe_backend='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
Process freetoken-TP0-scheduler:
[2026-08-31|00:36:47|FrontendAPI] ERROR Backend supervisor: DistNetworkError: The server socket has failed to listen on any local network address. port: 8082, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/user/dev/local-ai/upstream/FreeToken/python/freetoken/server/launch.py", line 81, in _run_scheduler
scheduler = Scheduler(args)
^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/python/freetoken/scheduler/scheduler.py", line 65, in __init__
self.engine = Engine(config)
^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/python/freetoken/engine/engine.py", line 309, in __init__
self.tp_cpu_group = self._init_communication(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/python/freetoken/engine/engine.py", line 430, in _init_communication
torch.distributed.init_process_group(
File "/home/user/dev/local-ai/upstream/FreeToken/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
func_return = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 1831, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/.venv/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 232, in _tcp_rendezvous_handler
store = _create_c10d_store(
^^^^^^^^^^^^^^^^^^^
File "/home/user/dev/local-ai/upstream/FreeToken/.venv/lib/python3.12/site-packages/torch/distributed/rendezvous.py", line 199, in _create_c10d_store
return TCPStore(
^^^^^^^^^
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8082, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
INFO: 127.0.0.1:56580 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56586 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56612 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56620 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56644 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56658 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56680 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56692 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56736 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56738 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56750 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56764 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56790 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56806 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56824 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56838 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56860 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56866 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56896 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:56906 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54380 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54390 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54408 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54418 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54436 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54440 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54448 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54450 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54490 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54500 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54524 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54534 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54556 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54558 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54572 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54576 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54594 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54606 - "GET /metrics HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54626 - "GET /props HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:54642 - "GET /metrics HTTP/1.1" 404 Not Found
[2026-08-31|00:36:57|FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API server
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [3049384]
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Suggested fix
An env override keeps the default behaviour and unblocks the collision
case; we run this locally:
@property
def distributed_addr(self) -> str:
port = int(os.environ.get("FREETOKEN_DIST_PORT", self.server_port + 1))
return f"tcp://127.0.0.1:{port}"A --dist-port flag may be the nicer long-term shape; happy to test either
on this machine.
Transparency (home directory in paths/logs anonymised to /home/user): this report was prepared with AI assistance; the failure and
the workaround were reproduced and verified on the hardware above.
Activity
benwilson commentedon Aug 31, 2026
One more reason a port override matters, and a reason it is not sufficient on its own:
distributed_addris declared astcp://127.0.0.1:{port+1}, but the socket that actually listens is bound to all interfaces.Environment: FreeToken git
4b94bdc(HEAD 2026-08-30, no local patches), installed withpip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA toolkit 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86) on a rented vast.ai host (503 GiB RAM); imagepytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel; checkpointdealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4@be794b99(standard compressed-tensors NVFP4 layout, byte-identical quant config to the RadixArk stock checkpoint — not the FTW repack).,ft serve --host 127.0.0.1 --port 1919/proc/net/tcp6inside the container showsLISTEN :::1920owned by the backend worker (multiprocessing.spawnchild offt serve), while the API itself sits on127.0.0.1:1919as asked. Inside Docker with only the API port published it is unreachable from outside (verified with a connect timeout), but on a bare host or--network hostit is an unauthenticated TCPStore on every interface regardless of--host. So alongside a port flag, the store should be created with the--hostaddress (or an explicit--distributed-hostdefaulting to 127.0.0.1) rather than the wildcard.ralfkuh-lab commentedon Sep 6, 2026
@benwilson Thanks for catching this — reproduced here on torch 2.11.0+cu130:
TCPStore("127.0.0.1", port, ...)still shows up inss -ltnpas*:port.One wrinkle for the fix shape: the host in the built-in
tcp://rendezvous URL is only what the clients dial; it does not control the store's bind address, so feeding--hostinto the URL would not give you the loopback bind. What does work is a pre-bound socket handed in viamaster_listen_fd(verified here, including with the libuv backend) — but that means constructing the store explicitly and passing it viastore=instead ofinit_method.Given that, I think the better fix is to drop the TCP rendezvous for the server launcher altogether. Every TP rank is a local
mp.Processchild offt serve(server/launch.py), so afile://rendezvous fits, and it removes both problems at the root: no port to collide with (this issue) and no TCPStore listener to expose (your finding).What I tested, CPU/Gloo only:
world_size=1, and separately two spawned local ranks —init_process_group,all_reduceandbroadcast_object_listall succeed with FileStore, with no store listener at all. This is not a multi-GPU run; the NCCL and PyNCCL TP paths inengine/engine.py::_init_communicationstill need validating on a machine that has them.Two implementation notes, so this does not end up as a one-line URL swap:
ServerArgs._unique_suffixis just.pid=…, so it is not a fresh-path guarantee across restarts. A privatetempfile.mkdtemp()created once in the parent (honouringTMPDIR), the same path handed to every rank, removed after the workers exit, is the shape I would go for.GLOO_SOCKET_IFNAME), and it says nothing about NCCL's network exposure.If the maintainers would rather keep TCP, an explicitly built store with a loopback-bound
master_listen_fdplus a configurable port is the working alternative — just not the host in the URL.YevheniiKotyrlo commentedon Oct 6, 2026
A single rank needs no rendezvous at all. #615 forms its group over a
FileStoreat a freshtempfile.mkstemppath and deletes the file as soon asinit_process_groupreturns. The group never reads the store again, so no port is bound, and a run killed after that point leaves no file behind. I held127.0.0.1:<port + 1>and started a Qwen3-0.6B serve:mainfails (EADDRINUSEon Linux; error 10048 on Windows, over the Windows PRs a serve needs there), and the change serves on both. More than one rank keeps the TCP rendezvous, where #461's loopback bind still applies; on Windows itsTCPStore(...)needsuse_libuv=False, since torch's Windows build has no libuv.