convert to guff and ollama

[Aniwine]

I want to deploy it via ollama, so I firstly convert it to .guff file by llama.cpp's convert_hf_to_guff.py,but I got an error that KeyError "<|user|>",so I found it not in added_tokens_decoder of tokenizer_config.json while it exists in convert_hf_to_guff.py.So I comment this line out, code can finish normally.And ollama create model also successfully completed.But when ollama run model,I got tensor number errors,

I wonder what happens after removing the <|user|> keyword and how should I fix this?

[ExtReMLapin]

Same issue.

[ExtReMLapin]

Working on it, managed to get a gguf working

[ExtReMLapin]

https://huggingface.co/LPN64/LongCite-llama3.1-8b-GGUF

[Aniwine]

https://huggingface.co/LPN64/LongCite-llama3.1-8b-GGUF

thank you, i want to know how did you solve this problem please?

[ExtReMLapin]

I used the gguf llamacpp conversion tool but it was missing files so I literally copy pasted files from llama 3.1 (non instruct model) and modified the end of sentence token in generation config file

[Aniwine]

I used the gguf llamacpp conversion tool but it was missing files so I literally copy pasted files from llama 3.1 (non instruct model) and modified the end of sentence token in generation config file

thank you~