How to integrate/run vLLM with XSched? (Missing example/docs)

[tonghaoxin]

Hi XSched team,

I am evaluating XSched on Alibaba Cloud ECS instances with NVIDIA A10 GPUs (Ampere architecture).

I have successfully verified that XSched works correctly with flash_attention workloads on my setup, which confirms that the basic hook/shim mechanism is functioning properly on the A10.

However, I am unable to get vLLM to work with XSched. I noticed in the README.md under the Development Plan, the item Integrated into LLM serving systems (e.g., llama.cpp, vLLM) is marked as completed [x].

My Issues:

1. Missing Integration Code: I can see dedicated directories for integration/llama.cpp and integration/triton in the source tree, but I cannot find any directory or examples for vLLM integration.
2. Scheduling Failure: When I try to run vLLM directly with XSched (using the transparent scheduling method), the scheduling does not seem to take effect or the engine fails to start properly.

Questions:

1. Is the vLLM integration guide or patch available? Could you please point me to the relevant branch or documentation?
2. Does vLLM require any specific "Level-2" or "Level-3" preemption features? I noticed that for Ampere GPUs (sm86), these levels are currently marked as "in progress" (🚧) in the support matrix.

Thanks for your amazing work!

[wuwen03]

Hi @tonghaoxin,

Thanks a lot for reporting these issues.

At the moment, vLLM is only supported through the transparent scheduling method. The current README.md is not accurate enough on this point. Support for vLLM, especially in the same dedicated integration style as llama.cpp, is still under investigation, and we will update the documentation to make this clearer.

Since you have already opened a PR related to this topic, if the transparent scheduling method still does not work on your setup, please feel free to share more details with us. Further discussion of the implementation itself can continue in the PR thread.

For your second question, vLLM does not require a specific preemption level. However, for Ampere GPUs (sm86), the currently supported options are Level-1 and Level-3 implemented via TSG. We will also clarify this in the documentation.