Prevent punctuation to be included in tokens #54
Description
Hello,
I have notice that if a work ends with a punctuation mark, the punctuation mark is included as as part of the token.
Is there a way to prevent this behaviour?
Best,
Ed
{{ message }}
Hello,
I have notice that if a work ends with a punctuation mark, the punctuation mark is included as as part of the token.
Is there a way to prevent this behaviour?
Best,
Ed