All labels get 0.99 as score

[stephantul]

Hello!

I am trying out gliclass. If I follow the example with a simple query, I see that every class label I pass gets 0.999 as a score. Is this expected?

labels = ["sports", "weather", "entertainment", "travel"]

results = pipeline(["weather"], labels, threshold=0.5)
print(results)
# [[{'label': 'sports', 'score': 0.9976825714111328},
#   {'label': 'weather', 'score': 0.9999980926513672},
#   {'label': 'entertainment', 'score': 0.9933828115463257},
#   {'label': 'travel', 'score': 0.9999979734420776}]]

pipeline was initialized as in the README.

Stéphan

[Ingvarstep]

Hello!

@stephantul , can you say which model do you use and your transformers version?

[stephantul]

Hello @Ingvarstep,

Apologies for not providing it before. The full script is this:

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

model = GLiClassModel.from_pretrained("knowledgator/gliclass-small-v1.0")
tokenizer = AutoTokenizer.from_pretrained("knowledgator/gliclass-small-v1.0")

pipeline = ZeroShotClassificationPipeline(model, tokenizer, classification_type='multi-label', device='cuda:0')

labels = ["sports", "weather", "entertainment", "travel"]

results = pipeline(["weather"], labels, threshold=0.5)
print(results)
# [[{'label': 'sports', 'score': 0.9976825714111328},
#   {'label': 'weather', 'score': 0.9999980926513672},
#   {'label': 'entertainment', 'score': 0.9933828115463257},
#   {'label': 'travel', 'score': 0.9999979734420776}]]

I can't find a short query that actually assigns any label a lower score (i.e., a score < 0.99 for any label). For example "watching tv" also gets very high scores.

Here's my environment. This behavior also appeared on an older version of transformers.

transformers==4.45.2
gliclass==0.1.7

Stéphan

[Ingvarstep]

Hello @Ingvarstep,

Apologies for not providing it before. The full script is this:

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

model = GLiClassModel.from_pretrained("knowledgator/gliclass-small-v1.0")
tokenizer = AutoTokenizer.from_pretrained("knowledgator/gliclass-small-v1.0")

pipeline = ZeroShotClassificationPipeline(model, tokenizer, classification_type='multi-label', device='cuda:0')

labels = ["sports", "weather", "entertainment", "travel"]

results = pipeline(["weather"], labels, threshold=0.5)
print(results)

[[{'label': 'sports', 'score': 0.9976825714111328},

{'label': 'weather', 'score': 0.9999980926513672},

{'label': 'entertainment', 'score': 0.9933828115463257},

{'label': 'travel', 'score': 0.9999979734420776}]]

I can't find a short query that actually assigns any label a lower score (i.e., a score < 0.99 for any label). For example "watching tv" also gets very high scores.

Here's my environment. This behavior also appeared on an older version of transformers.

transformers==4.45.2
gliclass==0.1.7

Stéphan

Hello @stephantul ,

Thank you for informing the issue, it is related to the loss function that was incorrectly set for such models. We gonna replace those models with fixed versions. However, they are still good for few-shot learning.

Ihor