How to deal with infinite loops #1
Description
With infinite loops, end users of a program that uses a library that uses tokenz would have no idea why their program is failing.
This is partially a responsibility of how users of tokenz code their tokenizing algorithm, but we could facilitate the checking of infinite loops.
Another way to deal with this is to add proper logging to stdout as a text is walked (for example tokenizing 0%... 100%). Because of this users would quickly know that the tokenization is not advancing. This could be a better solution and would result in better UI at the same time.