UserWarning coming from click library

[bcantoni]

ttok is showing a UserWarning coming from the click library reporting that --tokens is used more than once:

$ ttok --version
/Users/brian/.pyenv/versions/3.13.3/envs/llm-dev-3.13/lib/python3.13/site-packages/click/core.py:1193: UserWarning: The parameter --tokens is used more than once. Remove its duplicate as parameters should be unique.
  parser = self.make_parser(ctx)
/Users/brian/.pyenv/versions/3.13.3/envs/llm-dev-3.13/lib/python3.13/site-packages/click/core.py:1186: UserWarning: The parameter --tokens is used more than once. Remove its duplicate as parameters should be unique.
  self.parse_args(ctx, args)
ttok, version 0.3

I can see where this started with click 8.2.0 which included adding this warning: pallets/click#2396

The ttok utility still works fine. Downgrading to click 8.1.8 stops that warning.

[ndemou]

What I think is the root cause

Click is generating this warning because the parameter --tokens has been registered for two different actions (encode_tokens and as_tokens):

@click.option("encode_tokens", "--encode", "--tokens", is_flag=True, help="Output token integers")
@click.option("as_tokens", "--tokens", is_flag=True, help="Output full tokens")

Based on the examples from the help --tokens behavior aligns with as_tokens(Output full tokens) and not with that of encode_tokens (Output token integers) so we should remove --tokens from click.option("encode_tokens"

Here's the help:

To view token integers:

   cat input.txt | ttok --encode

[...]
To see the details of the tokens:

    ttok "hello world" --tokens

 Outputs:

    [b'hello', b' world'] 

The fix (if I guessed correctly above)

-@click.option("encode_tokens", "--encode", "--tokens", is_flag=True, help="Output token integers")
+@click.option("encode_tokens", "--encode",             is_flag=True, help="Output token integers")