Low AJ and Delta for DAVIS Strided mode evaluation

[YannickGibson]

Dear @nikitakaraevv,

The results for Tap-Vid DAVIS Strided mode are not listed in the paper, however, running the evaluation on my GPU (NVIDIA RTX 4000 Ada Generation) I am getting lower results than in the Query First mode. Is this on track with what results I should be getting? Results on Tap-Vid DAVIS Query First mode seem to check out with the paper results.

Thank you!

Overview

• Used weights scaled_online.pth
• Strided mode low results of AJ 48.5% and Average Delta of 58.8%
• Query First mode - AJ 64.5% where the paper has AJ 64.5% as well
• Query First mode - I get Average Delta of 77.2%, which is close to the 76.7% noted in README.md
• GPU: NVIDIA RTX 4000 Ada Generation
• Versions
  • Python 3.10.4
  • PyTorch: 1.13.0+cu117
  • NumPy: 1.26.4
  • Hydra: 1.1.0

Strided mode

Command

python ./cotracker/evaluation/evaluate.py --config-name eval_tapvid_davis_strided exp_dir=./eval_outputs/strided dataset_root=datasets

Strided Results JSON

{
    "occlusion_accuracy": 0.6789913433103425,
    "pts_within_1": 0.3681140514159171,
    "jaccard_1": 0.2810965209136301,
    "pts_within_2": 0.5160547528215811,
    "jaccard_2": 0.4269992312134806,
    "pts_within_4": 0.6188633490338561,
    "jaccard_4": 0.5340930209458256,
    "pts_within_8": 0.6852129258977692,
    "jaccard_8": 0.5834998947998223,
    "pts_within_16": 0.7527433967957153,
    "jaccard_16": 0.597554944076797,
    "average_jaccard": 0.48464872238991114,
    "average_pts_within_thresh": 0.5881976951929677,
    "time": 168.436683177948
}

Quer First mode

Command - Query First Evaluation:

python ./cotracker/evaluation/evaluate.py --config-name eval_tapvid_davis_first exp_dir=./eval_outputs dataset_root=datasets

First Results JSON

{
    "occlusion_accuracy": 0.9083404401790212,
    "pts_within_1": 0.4291642519990898,
    "jaccard_1": 0.2886003405266799,
    "pts_within_2": 0.6744320115565117,
    "jaccard_2": 0.5166554417691157,
    "pts_within_4": 0.8539922645434174,
    "jaccard_4": 0.725129847549515,
    "pts_within_8": 0.9343085429265164,
    "jaccard_8": 0.8273602850914099,
    "pts_within_16": 0.9693541925985442,
    "jaccard_16": 0.865224065618229,
    "average_jaccard": 0.6445939961109898,
    "average_pts_within_thresh": 0.7722502527248157,
    "time": 79.40006947517395
}

[Xueyangwang-will]

I ran into the same problem as you. Have you solved it?

[YannickGibson]

I ran into the same problem as you. Have you solved it?

No, I suppose it just performs worse in strided mode.

[Xueyangwang-will]

But the first-query mode doesn’t seem to match the metrics reported in the paper?

[YannickGibson]

That difference is small enough to be negligible.

[Xueyangwang-will]

The weights you used are scaled_online.pth. The officially reported AJ metric is 63.8, but the result you measured is 64.4. That’s quite a large discrepancy, isn’t it?

[YannickGibson]

So I'd say the first-query results are reproducing the paper, just with the small noise floor you'd expect. The strided-mode numbers are a separate question (the paper doesn't report them), and the gap there is much larger and worth a real explanation, but the first-query gap isn't.