Issues with running `model.generate()`

[Nadav-Nesher]

Hi guys,

I came across your model in a YouTube video and was excited to try it out!! 🚀

I'm trying to run the code from your Hugging Face page but running into some trouble.

Could you please assist?
Many thanks 🙏🏼

This is the code:

`pip install accelerate
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("kaist-ai/CoT-T5-3B")
model = T5ForConditionalGeneration.from_pretrained("kaist-ai/CoT-T5-3B", device_map="auto")

input_text = "Read the Directions and try to pick among A,B,C,D.\n\nDirecitons: A good way to figure out the relationship in a given question is to make up a sentence that describes the relationship between the first two words. Then, try to use the same sentence to find out which of the answer choices completes the same relationship with the third word.\nQuestion: Odometer is to mileage as compass is to?\nOptions: (A) speed, (B) hiking, (C) needle, (D) direction.\nLet's think step by step.\n"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
`

I also tried replacing the tokenizer and model with these:

1. When running the code with this tokenizer and model:

`from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("kaist-ai/CoT-T5-3B")
model = AutoModelForSeq2SeqLM.from_pretrained("kaist-ai/CoT-T5-3B")`

The was the error:
"""RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)"""

2. When running the code with this tokenizer and model:
   `from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("kaist-ai/CoT-T5-3B")
model = T5ForConditionalGeneration.from_pretrained("kaist-ai/CoT-T5-3B", device_map="auto")
`

This was the error:
"""C:\Users\user\anaconda3\python.exe C:\Users\user\PycharmProjects\playground\chain_of_thought.py
Traceback (most recent call last):
File "C:\Users\user\PycharmProjects\playground\chain_of_thought.py", line 4, in
model = T5ForConditionalGeneration.from_pretrained("kaist-ai/CoT-T5-3B", device_map="auto")
File "C:\Users\user\anaconda3\lib\site-packages\transformers\modeling_utils.py", line 2777, in from_pretrained
) = cls._load_pretrained_model(
File "C:\Users\user\anaconda3\lib\site-packages\transformers\modeling_utils.py", line 2871, in _load_pretrained_model
raise ValueError(
ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format."""

Note: I have safetensors installed.