Move tokenizer/tokenizer.py inside diaparser#2

Open

KoichiYasuoka wants to merge 4 commits into

Unipisa:masterfrom

KoichiYasuoka:patch-1

Commits on Dec 18, 2020

Commits on Mar 29, 2021

Update tokenizer.py to follow Stanza 1.2
KoichiYasuoka
authored