Should One Depthwise Convolution Be Frozen During Training?

[Haus226]

Thank you for the great work and new insight brought, however I have several questions outlined as below:

The paper states that during training, the model maintains two paths:

1. A fixed depthwise convolution with designed kernel weights to perform structured shifting.
2. A learnable depthwise convolution to enable more flexible feature shifts.

At inference, these two paths are merged into a single convolution via re-parameterization to maintain efficiency.

However, in the Slide-Transformer implementation, the first depthwise convolution (dep_conv) is not frozen during training. Instead, its parameters are updated through backpropagation.

This raises two key concerns:

• Contradiction with the paper: If dep_conv is meant to provide a fixed shift, why is it trainable? Shouldn't it be frozen during training?
• Necessity of Two Convolutions: If both convolutions are learnable, the shifting operation is no longer fixed. In this case, why not simplify the design and use a single depthwise convolution instead of two?

Clarification Needed

• Should dep_conv be explicitly frozen during training to match the paper's description?
• Or is the intended behavior actually a fully flexible shifting mechanism, meaning the two convolutions are both trainable from the start?