Add TIGER model for Semantic ID recommendation#711
Conversation
|
I've done some tuning with the three Amazon review datasets that are commonly used in Semantic ID papers. The table below shows
Setting: Reproducing Between the two configs, More detailed tuning, base encoder (ablation studies in
|
| Category | Recipe | Tokenizer | R@5 | N@5 | R@10 | N@10 | R@20 | N@20 | Train(s) |
|---|---|---|---|---|---|---|---|---|---|
| beauty | default | rqvae | 0.0217 | 0.0132 | 0.0386 | 0.0186 | 0.0591 | 0.0238 | 8072 |
| beauty | grid | rkmeans | 0.0340 | 0.0216 | 0.0564 | 0.0288 | 0.0885 | 0.0368 | 4772 |
| beauty | paischer | rqvae | 0.0419 | 0.0270 | 0.0634 | 0.0338 | 0.0953 | 0.0419 | 4341 |
| beauty | paischer | rkmeans | 0.0404 | 0.0259 | 0.0641 | 0.0335 | 0.1005 | 0.0427 | 9386 |
| sports | default | rqvae | 0.0157 | 0.0098 | 0.0262 | 0.0132 | 0.0442 | 0.0177 | 11075 |
| sports | grid | rkmeans | 0.0188 | 0.0117 | 0.0329 | 0.0162 | 0.0540 | 0.0215 | 5503 |
| sports | paischer | rqvae | 0.0230 | 0.0149 | 0.0373 | 0.0195 | 0.0596 | 0.0251 | 14102 |
| sports | paischer | rkmeans | 0.0240 | 0.0155 | 0.0394 | 0.0204 | 0.0635 | 0.0265 | 15917 |
| toys | default | rqvae | 0.0192 | 0.0115 | 0.0325 | 0.0158 | 0.0509 | 0.0204 | 6366 |
| toys | grid | rkmeans | 0.0298 | 0.0189 | 0.0506 | 0.0256 | 0.0780 | 0.0325 | 4270 |
| toys | paischer | rqvae | 0.0320 | 0.0205 | 0.0521 | 0.0269 | 0.0822 | 0.0345 | 9698 |
| toys | paischer | rkmeans | 0.0348 | 0.0222 | 0.0578 | 0.0295 | 0.0922 | 0.0382 | 9544 |
| toys | paischer +desc | rkmeans | 0.0362 | 0.0237 | 0.0583 | 0.0308 | 0.0915 | 0.0391 | 8609 |
A few other things to note:
TIGERlikely usesSentence-T5-XXLembedding, but improvements are minimal (sometimes even decrease)beam_searchdoesn't show improvements overexactscoringbeautyandsportsare somewhat reproducible, buttoysis not- Using OpenAI's
text-embedding-3-largeembedding (provided byRPG) might improve the performance, but not a lot - Well-tuned
SASRecandBERT4Rec(withincornac) still outperformTIGER. Quick runs from Global temporal and leave-last-out splitting for next-item evaluation #708 (comment) already showed we have better results thanSASRecandBERT4Rec's results fromTIGERpaper. I did some more detailed tuning forSASRecandBERT4Rec, and the results are much higher.
|
@qtuantruong @lthoang We do have all the tuning/experimental results from the notes above. Not sure if we should document it somewhere for reference? |
Description
This PR integrates
TIGER(Recommender Systems with Generative Retrieval) intocornac, along with dataset utilities to support its content-embedding requirements.Key Changes
load_text()tocornac/datasets/amazon_review.pyto parse product metadata (title, price, brand, categories, and description).tests/cornac/datasets/test_amazon_review.py.TIGERmodel implementation, usingNextItemRecommenderinterfacepaischer_config.pyandgrid_config.py) matching the settings used in popular benchmarks, to replicate results from the original paper.examples/tiger_example.pywith training pipeline on the Amazon Review (beauty) datasetRelated Issues
Checklist:
README.md(if you are adding a new model).examples/README.md(if you are adding a new example).datasets/README.md(if you are adding a new dataset).