Long-tailed Test-Time Adaptation for Vision-Language Models

This repository contains the code for Long-tailed Test-time Adaptation of Vision-Language Models.

⏳Setup

1. Environment

We test our codebase with PyTorch 2.1.1 with CUDA 12.1. Please install corresponding PyTorch and CUDA versions according to your computational resources. Then install the rest of required packages by running pip install -r requirements.txt. Please install the info-nce-pytorch package following info-nce-pytorch.

2. Datasets

Please kindly refer to the installation guidelines in DATASET.md to install all required datasets.

3. Replace the path

Use the global find/replace function in your IDE to:

1. Replace the /PATH/TO/YOUR/CODEBASE with the directory where you put this codebase.
2. Replace the /PATH/TO/YOUR/PRETRAIN with the directory where you would like to save the pretrained models.
3. Replace the /PATH/TO/YOUR/DATA with the directory where you put the above datasets.

📦Usage

You can refer to run.sh and change all of the arguments. The definition of these arguments are:

arguments	choices
--datasets	flowers102/caltech101/pets/cars/food101/aircraft/sun397/eurosat/ucf101/caltech101/A/R/V/S/I
--backbone	ViT_B/16,S,H,G
--nview	int, number of views
--imb	the reversed imbalance ratio.
--ctype	corruption type.
--severity	severity level of natural corruptions. 1-5.
--noise	severity of noise.

References

If you find this codebase helpful, please cite:

@inproceedings{wanglong,
  title={Long-tailed Test-Time Adaptation for Vision-Language Models},
  author={Wang, Xucong and Zhao, Zhe and Wang, Zekun and Cao, Xiaofeng and Wang, Xu and Wu, Di and Wang, Pengkun and Wang, Yang},
  booktitle={The Fourteenth International Conference on Learning Representations}
}