lists of integers instead of lists of tokenised words#22

Open

jwijffels wants to merge 26 commits intomasterfrom

Commits on Feb 19, 2024

Add test file

koheiw
authored and
jwijffels
committed
Change to serialized tokens

koheiw
authored and
jwijffels
committed
Move header file

koheiw
authored and
jwijffels
committed
Disable parameters for file inputs

koheiw
authored and
jwijffels
committed
Remove code for file inputs

koheiw
authored and
jwijffels
committed
Remove method for character

koheiw
authored and
jwijffels
committed
Change tokens from string to int

koheiw
authored and
jwijffels
committed
Remove vocabulary

koheiw
authored and
jwijffels
committed
Set frequency

koheiw
authored and
jwijffels
committed
Remove stopwords where types == ""

koheiw
authored and
jwijffels
committed
Don't remove any words

koheiw
authored and
jwijffels
committed
Disable save() and load()

koheiw
authored and
jwijffels
committed
Remove progress bar and callback for vocaburary

koheiw
authored and
jwijffels
committed
Improve handling of sentence lenghts

koheiw
authored and
jwijffels
committed
Use for loop

koheiw
authored and
jwijffels
committed
Clean up the code

koheiw
authored and
jwijffels
committed
Fix word and document index

koheiw
authored and
jwijffels
committed
Tidy up

koheiw
authored and
jwijffels
committed
Tidy up

koheiw
authored and
jwijffels
committed
Tidy up

koheiw
authored and
jwijffels
committed
Tidy up

koheiw
authored and
jwijffels
committed
Tidy up

koheiw
authored and
jwijffels
committed
Build

koheiw
authored and
jwijffels
committed
Update

koheiw
authored and
jwijffels
committed
Update

koheiw
authored and
jwijffels
committed
Update

koheiw
authored and
jwijffels
committed