Repository navigation
Add all available ONNX models to ORTConfigManager #351
Description
Activity
To update the list of supported models: BlenderBot, BLOOM, GptBigCode, GPT-NEOX, GPTJ, LongT5, Llama, mBART, M2M100, nystromformer, Pegasus,T5 ,ViT ,Whisper
@michaelbenayoun @fxmarty is there still interest in advancing with other models?Thanks @mszsorondo I just updated the list based on your comment 🙏 .
can i try to update for other model ?
- added a commit that references this issue
on Jul 20, 2026 #self-assign
I'd like to take the Flaubert subtask only. I plan to add the missing
NormalizedTextConfigmapping and a focused offline regression test covering the normalized hidden size, attention heads, and layer count.#self-assign LayoutLM subtask only. I will add the missing NormalizedConfigManager mapping and a focused offline test for hidden size, attention heads, and layer count.
- added a commit that references this issue
on Aug 21, 2026 #self-assign ConvBert subtask only.
I’d like to add the missing ConvBert configuration mapping to
ORTConfigManagerand the corresponding focused tests. Please let me know if this subtask is still available.#self-assign Data2VecText subtask only.
I'll add its missing NormalizedConfigManager mapping and a focused offline regression test covering normalized hidden size, attention heads, and layer count.
#self-assign mobilebert subtask only. I will add the missing NormalizedConfigManager mapping and a focused test.
#self-assign XLM subtask only.
I'll add XLM to
NormalizedConfigManager(the current equivalent ofORTConfigManager).XLMOnnxConfigalready exists in optimum-onnx, andXLMConfigusesemb_dim/n_heads/n_layersrather than the BERT-style names. I'll wire those attribute names and add a focused offline regression test.- added a commit that references this issue
on Sep 18, 2026 #self-assign Perceiver subtask only.
I'd like to add the missing Perceiver mapping to
NormalizedConfigManager. Perceiver usesd_latentsfor the latent self-attention representation andnum_self_attention_headsfor its self-attention head count, so I plan to add the corresponding normalized configuration mapping together with a focused offline regression test.I'll keep the PR scoped to the Perceiver subtask.
- added a commit that references this issue
on Sep 22, 2026 #self-assign PLBart subtask only.
I'd like to add the missing PLBart mapping to
NormalizedConfigManagertogether with a focused regression test.PLBartConfigexposes BART-style configuration attributes such asd_model,encoder_attention_heads, andencoder_layers, so I plan to verify the appropriate normalized configuration against the current Transformers implementation and keep the PR limited to PLBart support.Please let me know if this subtask is already being worked on.
#self-assign ConvNeXT subtask only. I'll add the missing NormalizedVisionConfig mapping to ORTConfigManager and a focused offline test covering image_size and num_channels.
#self-assign BEiT and LeViT subtasks. I'll add the missing NormalizedVisionConfig mappings to ORTConfigManager and focused offline tests covering image_size and num_channels for each, same pattern as ConvNeXT above.
#self-assign MobileViT and Hiera subtasks. I'll add the missing NormalizedVisionConfig mappings to ORTConfigManager and focused offline tests covering image_size and num_channels for each, same pattern as the ConvNeXT/BEiT/LeViT PRs above.
#self-assign ConvNeXtV2 and Data2VecVision subtasks. Same pattern as the ConvNeXT/BEiT/LeViT/MobileViT/Hiera PRs above: NormalizedVisionConfig mapping plus a focused offline test covering image_size and num_channels for each.
#self-assign DETR subtask. Its config resolves hidden_size/num_attention_heads/num_hidden_layers through its own attribute_map already, so plain NormalizedTextConfig works with no custom subclass -- adding the mapping plus a focused offline test.
#self-assign table-transformer subtask. Same situation as DETR (it's DETR-based) -- config resolves hidden_size/num_attention_heads/num_hidden_layers through its own attribute_map, so plain NormalizedTextConfig works, no custom subclass needed.
#self-assign LayoutLMv3 subtask. Config exposes hidden_size/num_attention_heads/num_hidden_layers directly (standard BERT-style naming), so plain NormalizedTextConfig works, no custom subclass needed. Note LayoutLMv2 does NOT have an ONNX config yet in this repo, so it's not actually eligible despite being on the checklist above.
#self-assign GroupViT subtask. Found a real bug along the way: NormalizedTextAndVisionConfig.getattr prefixes the raw accessor name onto the sub-config path instead of resolving it through the NUM_LAYERS-style mapping first -- e.g. .num_layers looks up a literal 'num_layers' attribute on text_config instead of 'num_hidden_layers'. This happens to work today only for Pix2Struct, because its text_config defines both spellings as synonyms; it silently breaks for GroupViT (and OwlViT/OwlV2/Siglip, same shape), whose sub-configs only have num_hidden_layers. Fixed the base class (regression-tested against Pix2Struct's existing values) and registered GroupViT via the now-correct Pix2StructNormalizedTextConfig.
This issue is linked to the ONNXConfig for all working group created for implementing an ONNXConfig for all available models. Let's extend our work and try to add all models with a fully functional ONNXConfig implemented to ORTConfigManager.
Adding models to ORTConfigManager will allow 🤗 Optimum users to boost even more their model with ONNX optimization capacity!
Feel free to join us in this adventure! Join the org by clicking here
Here is a non-exhaustive list of models that have one ONNXConfig and could be added to ORTConfigManager:
This includes only models with ONNXConfig implemented, if your target model doesn't have an ONNXConfig, please open an issue/or implement it (even cooler) in the 🤗 Transformers repository. Check this issue to know how to do
If you want an example of implementation, I did one for
MT5#341.You need to check how the
attention_headsnumber andhidden_sizearguments are named in the original implementation of your target model in the 🤗 Transformers source code. And then add it to the_confdictionary. Finally, add your implemented model to tests to make it fully functional.