Out of resources error

[csaiedu]

Getting the following
"OutOfResources: out of resource: shared memory, Required: 262144, "

CPU
total used free shared buff/cache available
Mem: 98607700 6376796 82308864 181960 9922040 91058404
Swap: 2097148 0 2097148
GPU

0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |
| 0% 39C P8 8W / 370W | 1080MiB / 24576MiB | 0% Default |
| | | N/A

Particularly interested in the results on GliClass with a few hundreds classes. So far at zero-shot the class distinction with knowledgator/gliclass-modern-large-v2.0-init hasn't shown great results (no fine-tuning)

[Ingvarstep]

Hi @csaiedu , thanks for testing it on RTX 3090, I didn't test FlashDeBERTa on this GPU, it may require more tuning of kernel parameters.

Regarding using GLiClass for a few hundred classes, I can suggest trying to use the labels chunking pipeline, because it's usually hard for a model to extrapolate for such a number of classes out-of-the-box.

https://github.com/Knowledgator/GLiClass/blob/19bf6a124a8fe9a81720591cf9980c5870738a82/gliclass/pipeline.py#L287

[csaiedu]

Thank you for the suggestions. GliClass is struggling to distinguish even with the chunking pipeline.

All the best

[4rste1f]

Hey there!
@csaiedu , could you please specify the pipeline your are currently using so that we can reproduce the error more reliably? Is it the test example from the repo's main page or something else?

[csaiedu]

Yes standard example tested on both cpu and cuda

from flashdeberta import FlashDebertaV2Model
from transformers import AutoTokenizer #4.51.3'
import torch   #'2.6.0+cu124'
import accelerate


device='cuda:0'

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base",device=device)
model = FlashDebertaV2Model.from_pretrained("microsoft/deberta-v3-base").to(device)
# Tokenize input text
input_text = "Hello world!"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)

# Model inference
# The above code is using a model to make predictions on the input_ids data and storing the outputs in
# the variable "outputs".
outputs = model(input_ids)

[vrdn-23]

Can confirm that this is reproducible for sequences of short length on a A10G (24GB) instance. It works on an L4, by pure luck because it doesn't match the CUDA compatibility checks and get assigned the lowest block configuration.
cc @Ingvarstep

[vrdn-23]

Is this issue going to be fixed or is it on anyone's radar? The library is really great work but this is virtually unusable for the most widely available consumer specific GPUs on AWS (g5 instances) because of the issue in calculating the shared memory values.
Is it possible to either make a fix or review the changes in #4?
cc @Ingvarstep

I would be happy to lend my time/efforts if this the solution in the PR is not adequate and it needs an extra set of hands on it