I currently use UnslothStudio (basically llama.cpp) through opencode, so I have a couple of options.
I can set up a wrapper so that the unsloth server opencode spawns also gets shut down when opencode is closed down. This is a bit of a pain to set up, and I haven't gotten it working yet with openChamber, but it does work.
Easier option is just to set up the auto idle timer in unsloth, if the model is idle for X time it will be unloaded from vram.
This is useful because I don't want my vram to be filled by an llm model 24/7. It would be annoying. And I don't want to have to manually go and shut down processes in a terminal to recover it.
I'm tempted to try ninfer for the extra speed on my 5090 though. So figured i'd ask the question and hope there's a method I'm not seeing!
I currently use UnslothStudio (basically llama.cpp) through opencode, so I have a couple of options.
I can set up a wrapper so that the unsloth server opencode spawns also gets shut down when opencode is closed down. This is a bit of a pain to set up, and I haven't gotten it working yet with openChamber, but it does work.
Easier option is just to set up the auto idle timer in unsloth, if the model is idle for X time it will be unloaded from vram.
This is useful because I don't want my vram to be filled by an llm model 24/7. It would be annoying. And I don't want to have to manually go and shut down processes in a terminal to recover it.
I'm tempted to try ninfer for the extra speed on my 5090 though. So figured i'd ask the question and hope there's a method I'm not seeing!