Error while inferencing

[ra-MANUJ-an]

Hi @avidale While using the gedi_inference notebook, I'm experiencing the following error:

AttributeError: 'GediAdapter' object has no attribute 'can_generate'

Is there a solution for how to work around this problem?

I'm using : from transformers import GenerationMixin
instead of : from transformers.generation_utils import GenerationMixin

Complete error log is as follows:

====BEFORE====
The internal policy of Trump is flawed.
====AFTER=====
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File <timed exec>:7

File ~/second/lib/python3.11/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File ~/second/lib/python3.11/site-packages/transformers/generation/utils.py:1408, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1405         synced_gpus = False
   1407 # 1. Handle `generation_config` and kwargs that might update it, and validate the `.generate()` call
-> 1408 self._validate_model_class()
   1410 # priority: `generation_config` argument > `model.generation_config` (the default generation config)
   1411 if generation_config is None:
   1412     # legacy: users may modify the model configuration to control generation -- update the generation config
   1413     # model attribute accordingly, if it was created from the model config

File ~/second/lib/python3.11/site-packages/transformers/generation/utils.py:1184, in GenerationMixin._validate_model_class(self)
   1179 def _validate_model_class(self):
   1180     """
   1181     Confirms that the model class is compatible with generation. If not, raises an exception that points to the
   1182     right class to use.
   1183     """
-> 1184     if not self.can_generate():
   1185         generate_compatible_mappings = [
   1186             MODEL_FOR_CAUSAL_LM_MAPPING,
   1187             MODEL_FOR_CAUSAL_IMAGE_MODELING_MAPPING,
   (...)
   1190             MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING,
   1191         ]
   1192         generate_compatible_classes = set()

AttributeError: 'GediAdapter' object has no attribute 'can_generate'

[avidale]

Hi @ra-MANUJ-an!
I never had this bug before.
The last version of the transformers package that I tried was transformers==4.11.1, and the code worked fine with it.
What versions of Python and transformers are you using?

[ra-MANUJ-an]

For transformers I'm using transformers==4.35.0.dev0. I'll try to work with transformers==4.11.1 and update. If I want to use the latest version, where should I make the changes in the adapter code for generation? Is it possible to adapt for latest transformers version? For Python, my version is Python 3.11.6.

[avidale]

If I want to use the latest version, where should I make the changes in the adapter code for generation?

The code that interacts with the transformers internal API is in https://github.com/s-nlp/detox/blob/main/emnlp2021/style_transfer/paraGeDi/gedi_adapter.py, so you'll probably have to modify it (maybe, just by adding the can_generate() method to GediAdapter)

[avidale]

Is it possible to adapt for latest transformers version?

It is possible, but to be honest, I don't want to bother with this. The intent of this repository was just to make the results of my paper reproducible, not to create a reusable module and support it in the long term.

And if you really really want to make ParaGeDi reusable, I would recommend re-implementing it directly within https://github.com/huggingface/transformers; this would automatically solve the compatibility issue.

[ra-MANUJ-an]

The intent of this repository was just to make the results of my paper reproducible, not to create a reusable module and support it in the long term.

Oh sure! I understand. I was seeking to take inspiration from your code to reuse it for my application. By re-implementing, you mean working from scratch or change the functions?

[ra-MANUJ-an]

I have one more issue, not concerning this in particular but while installing transformer version 4.11.1 library. Did you encounter it before? It seems like Python library error to me ...

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 10.8 MB/s eta 0:00:00
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 14.0 MB/s eta 0:00:00
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 880.6/880.6 kB 19.1 MB/s eta 0:00:00
 Preparing metadata (setup.py) ... done
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.7/212.7 kB 18.7 MB/s eta 0:00:00
 Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
 error: subprocess-exited-with-error
 
 × Building wheel for tokenizers (pyproject.toml) did not run successfully.
 │ exit code: 1
 ╰─> See above for output.
 
 note: This error originates from a subprocess, and is likely not a problem with pip.
 Building wheel for tokenizers (pyproject.toml) ... error
 ERROR: Failed building wheel for tokenizers
 Building wheel for sacremoses (setup.py) ... done
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

[avidale]

No, I never had this issue.
My guess is that the version of tokenizers required by transformers==4.11.1 is not compatible with Python 3.11.
Or maybe their pyproject.toml is just misconfigured.
Anyway, I think this should be asked in the hugginface repo.

[avidale]

By re-implementing, you mean working from scratch or change the functions?

By re-implementing, I mean working from scratch, because an "ideal" reusable implementation should be much different from mine:

• The class inheriting from PreTrainedModel and working as a PyTorch module, so that one could, for example, fine-tune it with propagating gradients through parameters of any model (paraphraser, discriminator, or both).
• An interface general enough so that any text generation models can be plugged into it, e.g. any combination of decoder-only models (like GPT) or encoder-decoder models (like T5)

[ra-MANUJ-an]

By re-implementing, I mean working from scratch, because an "ideal" reusable implementation should be much different from mine:

• The class inheriting from PreTrainedModel and working as a PyTorch module, so that one could, for example, fine-tune it with propagating gradients through parameters of any model (paraphraser, discriminator, or both).
• An interface general enough so that any text generation models can be plugged into it, e.g. any combination of decoder-only models (like GPT) or encoder-decoder models (like T5)

I hope this is only being said for the gedi_adapter which is the main interface and where fusion of model happens.
Did you change beam_search function?

[ra-MANUJ-an]

hi @avidale will it be possible if you can try running the inference notebook at your end with transformers==4.11.1 and see if it works, thanks!

[avidale]

Did you change beam_search function?

No, I didn't, because I wanted to be able to use all the generation parameters already implemented in transformers.

Instead, I inherited GediAdapter from GenerationMixin that implements all the generation functions, so I had only to implement three methods that were used in these functions: __call__, prepare_inputs_for_generation, and _update_model_kwargs_for_generation.

[avidale]

hi @avidale will it be possible if you can try running the inference notebook at your end with transformers==4.11.1 and see if it works, thanks!

Actually, now I am getting the same installation error as you, Could not build wheels for tokenizers. Never had it before.

I'll dig into it later when I have time.

[ra-MANUJ-an]

hi @avidale will it be possible if you can try running the inference notebook at your end with transformers==4.11.1 and see if it works, thanks!

Actually, now I am getting the same installation error as you, Could not build wheels for tokenizers. Never had it before.

I'll dig into it later when I have time.

ok sure! let me know if it works for you

[ra-MANUJ-an]

Did you change beam_search function?

No, I didn't, because I wanted to be able to use all the generation parameters already implemented in transformers.

Instead, I inherited GediAdapter from GenerationMixin that implements all the generation functions, so I had only to implement three methods that were used in these functions: __call__, prepare_inputs_for_generation, and _update_model_kwargs_for_generation.

Yes, I was looking at the implementation and of GenerationMixin, I'll try to implement on the same lines. And just to clarify once, this method can be used for current transformer's class too, right?

[avidale]

@ra-MANUJ-an
After installing transformers==4.24, all the other requirements did install well, and I was able to reproduce my previous generations in this Colab notebook

Thus, I recommend you to switch to this version of transformers, and hopefully this will resolve your issue.