[Question]how do you seperate the onnx into multiple file?

[twmht]

• The question is written in English (I can translate what you say, but questions written in English are easier to read for the other users).

Write here the question.

Could you please explain how you split the ONNX model on Olive into multiple files? What were the guiding principles or your experience behind the splitting?

[niedev]

Hi, this has nothing to do with Olive, here is a superficial explanation on how and why I separated the models. To do the separation I used a python script based on this tutorial.

[Anderhar]

Looks like this request is gaining popularity even outside the RTranslator community. Personally, I tried to follow the linked tutorial to convert model for Whisper+ IME (because it utilizes your models too), but failed to get through the barrage of errors produced by modify_convnet.py script.

The need to do this is obvious when you speak a language other than English: unfortunately, small Whisper is too weak to really perform well in a multilingual context, making localized models still indispensable.

@niedev , I know you prefer to focus your efforts on improving RTranslator, but the truth is that sideloading the optimal model can make the app much more useful and valuable than some other improvements. Please reconsider the priority of creating a comprehensive conversion guide (because it seems that you are the only one who understands this process well enough).

[niedev]

Hi @Anderhar, I understand your issue, and thanks for your explanation.

The problem is that, although these optimizations principles can be applied in every transformer model, every model requires a different way to do it (that's why I can't do a conversion script) because they have different structures, and the same principle applies to model inference (if you look at my inference code you can see that with whisper, nllb and madlad I use different logic). That's the reason why a detailed guide would be very complicated and long to write and it would be very complicated and long to follow (not much easier than the short one I already made).

So this guide would not be used to sideload models, it will probably be used only by other developers (and they are very few). Plus I have a 9 to 6 job (I have very very few donations from this project) and in this period I can't even find the time to work on the app.

I'm not saying that I will never do a detailed guide, if one day I will have enough free time I will be happy to do it. But for now the effort would not match the outcome (not only for me, as I said, this guide would befit very few people, maybe 10), unlike other improvements I can make to the app.

[Anderhar]

Now it seems I understand your motivation perfectly, @niedev . So let's approach it from a different angle: how about just adding support for unoptimized single-file .onnx models?

Before attempting to do the conversion, I naturally tried to import raw .onnx model (unsuccessfully, of course), but if I had succeeded in that, I wouldn't even bother you here.

If the device is powerful enough, it's easier to sacrifice optimization for adequate speech recognition with localized model.

[niedev]

@Anderhar I agree on the fact that sideloading unoptimized models is better than nothing, but the problem is that even in that case, every model has different inputs and pre processing requirements (for example different tokenizers, different methods to indicate the source and target languages, different inputs matrices sizes and formatting, ecc.).
I will try in the future to make an inference logic that can automatically adapt to different models, because I also think sideloading custom models has a lot of potential. But I don't know if it would be possible, certainly it will be hard. Anyway, thank you for your suggestions, I have added the support of sideloading to my list of things to do and I will try in the future 🚀

[Anderhar]

Thank you for your great responsiveness! I am very glad that my thoughts have at least been brought to your attention.

And speaking of Madlad: choosing from more than one pre-built model (Madlad, Parakeet, MMS, etc.) could also be a “better than nothing” solution.

[niedev]

Thank you!

[niedev]

Hi @twmht @Anderhar, it is not a proper guide, but I released the code used to optimize and run the models here! For now it only officially supports Madlad, HY-MT and TranslateGemma, but in the "Other" folder, inside madlad_modifier.py and enhancer_nllb.py, I kept all the old code used to optimize Whisper and NLLB. And the official support for these one will come in the future.

[Anderhar]

Madlad, HY-MT and TranslateGemma

Looks promising (except for that 64GB part, of course), but what about TTS models for voice generation? Seems like they're a bit out of context, right?

[niedev]

but what about TTS models for voice generation? Seems like they're a bit out of context, right?

I don't understand what you mean 😅 RTranslator doesn't use models for TTS, but the system TTS engine.

[Anderhar]

RTranslator doesn't use models for TTS

Pardon me: I'm also fell out of context. Naturally, I meant STT for voice input (i.e. Whisper type). That was my initial point of interest back in August and to this day.

[niedev]

Oh ok, as I said in the "Other" folder, inside madlad_modifier.py (I know, it is a stupid name), there are all the function necessarily to create the various components of Whisper, if you want you can play with those to create the version you wanted (no need to modify the functions internally, it is much simpler than following the short guide I've linked at the beginning of the issue).
However, in the next release of RTranslator (or the one after that) I will improve the quality of the model (via better quantization, if possible) and add Whisper support to OnnxModelsEnhancer, so there is also the option to wait (6 months to a year, but who knows, maybe sooner).
I just wanted to inform everyone that now a pseudo guide exists.