Popular repositories Loading

1. SaGe SaGe Public

   Code for SaGe subword tokenizer (EACL 2023)

   Python 28 7

2. tokenizers_intrinsic_benchmark tokenizers_intrinsic_benchmark Public

   Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"

   Python 13 1

3. Splintering Splintering Public

   Splintering Nonconcatenative Languages for Better Tokenization

   Python 3 1

4. MeLeL-BiVERT MeLeL-BiVERT Public

   MT evaluation - Carinne Thesis

   Jupyter Notebook

5. Privacy-Preserving-Token-Manipulation Privacy-Preserving-Token-Manipulation Public

   Python

6. Dchi-Stencil Dchi-Stencil Public

   Python