Letter-precise html tokenization#49

Merged

kmike merged 31 commits intoscrapinghub:masterfrom

whalebot-helmsman:master

Oct 2, 2017

Commits on Sep 22, 2017

letter perfect detokenization
whalebot-helmsman
committed

Commits on Sep 25, 2017

do not cleanup tokenized tree by default, separate method for tree cleanup
Vostretsov Nikita
committed
update tests for separate tree cleaning
Vostretsov Nikita
committed
update tests for correct punctuation positions
Vostretsov Nikita
committed
correct length for replaced quotes
Vostretsov Nikita
committed

Commits on Sep 29, 2017

pep8
Vostretsov Nikita
committed
comma at line end, not start
Vostretsov Nikita
committed
one join instead of many additions, dont be Schleimel
Vostretsov Nikita
committed
correct formatting
Vostretsov Nikita
committed
add clarification
Vostretsov Nikita
committed
fix typo
Vostretsov Nikita
committed
pep8
Vostretsov Nikita
committed
preserve tokenize method for compatibility
Vostretsov Nikita
committed
function to reduce code in tests
Vostretsov Nikita
committed
remove test for nltk tokenizer
Vostretsov Nikita
committed
test our behaviour, which difers from original treebank tokenizer
Vostretsov Nikita
committed
remove useless conversion
Vostretsov Nikita
committed
rename method to avoid confusion with nltk tokenize_span method
Vostretsov Nikita
committed
remove brittle tests
Vostretsov Nikita
committed
small benchmark for html tokenizer
Vostretsov Nikita
committed

Commits on Oct 2, 2017