Preserve token strategy used at inference stage

[King4819]

Hi, I want to ask that what is the strategy at inference stage ? The method utilizes gumbel softmax to generate preserve token mask at training stage, but how to generate preserve token mask at inference stage ?

[ZLKong]

For the inference stage, we remove (hard prune) the tokens for speedup.

SPViT/vit_l2_3keep_senet_inference.py

Line 563 in 7ff446d

x = batch_index_select(x, now_policy)

We only use gumbel softmax during training. The same approach is used in DynamicViT

https://github.com/raoyongming/DynamicViT/blob/1322e626d1e9eca18cc150569908fb4af29e15f7/models/dyvit.py#L449

[King4819]

@ZLKong Thanks for your reply. If during inference stage it utilizes the technique same as DynamicViT, how do it achieve adaptive pruning? Since the preserve number of tokens is already determined

[King4819]

@ZLKong Sorry. I'm wondering if you have seen the reply, thanks !!!