Skip to content

Add extra tests for cjk#49

Open
pickfire wants to merge 1 commit intowithoutboats:masterfrom
pickfire:cjk
Open

Add extra tests for cjk#49
pickfire wants to merge 1 commit intowithoutboats:masterfrom
pickfire:cjk

Conversation

@pickfire
Copy link

@pickfire pickfire commented Aug 16, 2023

I am not sure if the failures are correct.

@jplatte
Copy link
Collaborator

jplatte commented Aug 22, 2023

The failing tests all have fullwidth punctuation that does not survive filtering of punctuation but counts as a word break. I would tend to agree with the test as written failing, why would fullwidth punctuation not be stripped + count as word breaks?

Also, tests 16 and 18 seem to be identical and they're not numbered consistently (e.g. 17 is missing).

@pickfire
Copy link
Author

Also, tests 16 and 18 seem to be identical and they're not numbered consistently (e.g. 17 is missing).

16 is simplified chinese, 17 (previously 18) is traditional chinese.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants