0.3

• New --allow-special option for allowing special tokens: ttok '<|endoftext|>' --encode --allow-special #13

0.2

• --encode now encodes text to integer tokens and displays them. This has been renamed from --tokens.
• --decode against a sequence of integer tokens now turns those back into text. #7
• --tokens for either of those options outputs the raw tokens in a format that helps show if they have a leading space or similar. #4

0.1

• Initial release. echo input.txt | ttok to count tokens, use -t 100 to truncate to 100 tokens, or --tokens to see what those integer tokens would be. #1