Feature/divide split and tokenize - #552

Merged

MartinKl merged 17 commits into

mainfrom

feature/divide-split-and-tokenize

Aug 20, 2026

Commits on Aug 19, 2026

only clean in merge mode
MartinKl
committed
ea256ac
update snapshots
MartinKl
committed
652cb13
add split mode
MartinKl
committed
c367800
refactoring, no inplace operations
MartinKl
committed
980ef2e
increase visibility of tokenizer
MartinKl
committed
a282647
increase visibility of tokenizer
MartinKl
committed
d73e67a
increase visibility of tokenizer
MartinKl
committed
fa0ab10

Commits on Aug 20, 2026

new error type
MartinKl
committed
07ebf0c
increased visibility
MartinKl
committed
00de617
allow to divide using treetagger tokenizer
MartinKl
committed
a61c7b5
new snapshots
MartinKl
committed
3db5886
new test data
MartinKl
committed
7520543
update
MartinKl
committed
bf4630d
update test
MartinKl
committed
5a3169d
update
MartinKl
committed
9c9a150
a lot of homework from clippy
MartinKl
committed
7435ffa
update
MartinKl
committed
fdf8e95