`pad_token_id` setting in generation_config

[hwq0726]

Hi,

I encountered a bug while using the "Generate watermarked output" section in a Colab notebook. After loading the model and immediately calling model.generate, the following error will be raised:

ValueError: stopping_criteria is not empty, pad_token_id must be set in generation_config.

Here is the snippet from your Colab notebook that reproduces the issue:
model = load_model(MODEL_NAME, expected_device=DEVICE, enable_watermarking=True)
torch.manual_seed(0)
outputs = model.generate( **inputs, do_sample=True, temperature=0.7, max_length=1024, top_k=40, )

The pad_token_id in the config is set to 0 which will return true when if not pad_token_id. I add this line to fix this:
model.generation_config.pad_token_id = model.generation_config.eos_token_id
This seems to resolve the error, but it appears that the default setting or handling of pad_token_id might need review.

BTW, this error did not occur a few days ago, maybe something changed when fixing Python compatibility?

Could you please look into this? Thanks!

[RyanMullins]

Thanks for the bug report, @hwq0726!

This ValueError was added in #6 to address issues that some users ran into adapting this to Llama 3 models, but you're right that the current check is potentially incorrect. I've changed the check in #13 to only raise if pad_token_id is None. Will merge, tag, and push a release as soon as the CI runs finish.

[RyanMullins]

Released as https://pypi.org/project/synthid-text/0.2.1/

[LOVEANDNINENINE]

When I tested this work on Colab, I noticed that the default handling of pad_token_id still seems to have some unresolved issues. This problem persists.

[RyanMullins]

Can you be more specific?

[hwq0726]

When I tested this work on Colab, I noticed that the default handling of pad_token_id still seems to have some unresolved issues. This problem persists.

I think the issue relates to gpt2 model implementation. I also met this issue when using the gpt2 model on a CPU. By default, the pad_token_id in the generation configuration for gpt2 is set to None, which leads to this error.

[LOVEANDNINENINE]

When I tested this work on Colab, I noticed that the default handling of pad_token_id still seems to have some unresolved issues. This problem persists.

I think the issue relates to gpt2 model implementation. I also met this issue when using the gpt2 model on a CPU. By default, the pad_token_id in the generation configuration for gpt2 is set to None, which leads to this error.

Yes, this issue still exists when gpt2 is used, regardless of whether it's on CPU or GPU. It's actually not a big deal. You just need to add a line for: pad_token_id value in default setting. @RyanMullins

[RyanMullins]

Thanks for following up, @LOVEANDNINENINE.

Looks like pad_token_id isn't set in in the GPT-2 generation config so it's probably raising a ValueError for that model.

huggingface/transformers#2630 suggests that this is a somewhat contested issue for GPT-2 and that submitting a PR to update the generation config likely won't be successful. I'll update the Colab to set this for you automatically if using GPT-2.