Inquiry about Binary (All-or-Nothing) Compression Behavior with the TQA Model

[AmmyQi]

Hello,
Thank you for making your model and code available. I have been testing the model (created by merging your Hugging Face adapter with the base model) using your code repository on a subset of the TQA dataset (dev.json, top 50 documents, ~5000 tokens,Llama3.1-8B-Instruct).
The overall results are approximately consistent with those reported in the paper:
{
"compressor": "exit",
"dataset": "TQA",
"dataset_type": "standard",
"num_samples": 500,
"successful_samples": 500,
"failed_samples": 0,
"avg_em": 0.564,
"avg_f1": 0.6387630837277899,
"avg_compression_ratio": 0.15294212103817084,
...
}

However, during a case-by-case analysis, I observed a specific behavior: for each example, the model tends to either retain almost all sentences or discard almost all sentences. It rarely performs fine-grained, selective compression.
I would like to ask if you have encountered this "binary" or "all-or-nothing" compression behavior before? If so, I would be very grateful for any insights into its potential causes and any suggestions for mitigating it.
Thank you for your time and consideration.

[AmmyQi]

Top5doc:
🚀 logprobs_dict: {1294: Logprob(logprob=-0.4342455267906189, rank=1, decoded_token='No'), 3553: Logprob(logprob=-1.0436205863952637, rank=2, decoded_token='Yes'), 651: Logprob(logprob=-11.074870109558105, rank=3, decoded_token='The'), 1307: Logprob(logprob=-11.199870109558105, rank=4, decoded_token='▁No'), 1261: Logprob(logprob=-11.926432609558105, rank=5, decoded_token='Answer'), 6287: Logprob(logprob=-12.059245109558105, rank=6, decoded_token='▁Yes'), 956: Logprob(logprob=-12.152995109558105, rank=7, decoded_token='no'), 21404: Logprob(logprob=-12.168620109558105, rank=8, decoded_token='Sure'), 1596: Logprob(logprob=-12.363932609558105, rank=9, decoded_token='This'), 3276: Logprob(logprob=-12.645182609558105, rank=10, decoded_token='yes'), 1718: Logprob(logprob=-13.488932609558105, rank=11, decoded_token='It'), 688: Logprob(logprob=-14.070963859558105, rank=12, decoded_token=''), 5036: Logprob(logprob=-14.336588859558105, rank=13, decoded_token='True'), 12156: Logprob(logprob=-14.399088859558105, rank=14, decoded_token='Sorry'), 5349: Logprob(logprob=-14.453776359558105, rank=15, decoded_token='NO'), 94638: Logprob(logprob=-14.645182609558105, rank=16, decoded_token='Certainly'), 3493: Logprob(logprob=-14.856120109558105, rank=17, decoded_token='There'), 17480: Logprob(logprob=-15.067057609558105, rank=18, decoded_token='YES'), 5958: Logprob(logprob=-15.090495109558105, rank=19, decoded_token='Please'), 235280: Logprob(logprob=-15.188151359558105, rank=20, decoded_token='A')}
But Top50doc:
🚀 logprobs_dict: {651: Logprob(logprob=-1.585160493850708, rank=1, decoded_token='The'), 2169: Logprob(logprob=-2.208207368850708, rank=2, decoded_token='As'), 886: Logprob(logprob=-2.655472993850708, rank=3, decoded_token='In'), 10142: Logprob(logprob=-2.719926118850708, rank=4, decoded_token='However'), 235248: Logprob(logprob=-2.987504243850708, rank=5, decoded_token='▁'), 1636: Logprob(logprob=-3.426957368850708, rank=6, decoded_token='For'), 1596: Logprob(logprob=-3.581254243850708, rank=7, decoded_token='This'), 10811: Logprob(logprob=-3.737504243850708, rank=8, decoded_token='While'), 1860: Logprob(logprob=-3.874222993850708, rank=9, decoded_token='But'), 688: Logprob(logprob=-4.056839942932129, rank=10, decoded_token=''), 2495: Logprob(logprob=-4.083207130432129, rank=11, decoded_token='If'), 2079: Logprob(logprob=-4.293168067932129, rank=12, decoded_token='And'), 235278: Logprob(logprob=-4.295121192932129, rank=13, decoded_token='('), 30021: Logprob(logprob=-4.306839942932129, rank=14, decoded_token='Despite'), 235287: Logprob(logprob=-4.353714942932129, rank=15, decoded_token='*'), 1718: Logprob(logprob=-4.499222755432129, rank=16, decoded_token='It'), 2122: Logprob(logprob=-4.556839942932129, rank=17, decoded_token='On'), 1469: Logprob(logprob=-4.584183692932129, rank=18, decoded_token='To'), 235280: Logprob(logprob=-4.624222755432129, rank=19, decoded_token='A'), 24160: Logprob(logprob=-4.779496192932129, rank=20, decoded_token='Therefore')}