Uniform predictions (identical saliency scores) during TVSum dataset evaluation with Lighthouse pipeline

[Andzy324]

When evaluating the Lighthouse model on the TVSum dataset using the provided scripts, the model outputs identical highlight scores across all time windows, despite following the tutorial steps.
eval_command:
python /lighthouse/training/evaluate.py --model uvcom --dataset tvsum --feature i3d_clip --domain BK --split val --model_path checkpoints/results/uvcom/tvsum/i3d_clip/BK/best.ckpt --eval_path data/tvsum/tvsum_val_release.jsonl
I printed out the saliency score predictions during the evaluation, and a screenshot is attached below to show the output.

1.Are there known issues with TVSum evaluation in the current version?

2.Could the problem stem from data loading or post-processing steps or shape problem of loaded features?
However, i downloaded all the checkpoints and extracted-features from the link on lighthouse repo page.
Appreciate your guidance to resolve this issue.

[awkrail]

@Andzy324
Thank you for reporting the issue.

1. Interesting. This problem is not known. Actually, current HD datasets (TVSum and Youtube Highlight) are too small for both training and evaluation, making it difficult to stabilize the training. I reproduced the scores to the original papers, but am not sure that such corrputed outputs. I will re-train other models with the original repo again, and check the model's saliency scores.

2. No, because I3D+text features are downloaded from CG-DETR repo. I reproduced the experiments based on them first, then incorporated the codes into Lighthouse. So, I think this problem stems from limited training/evaluation samples, rather than the features.

[Andzy324]

Thank you for your prompt response!
Could you provide your intermediate raw predictions of saliency scores, namely accessing preds = outputs['saliency_scores'] of your experiments. Besides, have you ever paid attention to the numeric predictions of UVCOM instead of mAP metric when running experiment on TVSum dataset? Since i have noticed that in the original implementation of UVCOM evaluation, the author did not provide direct access to the prediction results but only save the metrics. Using lighthouse to conduct evaluation on QVHighlight, i can get non-uniform saliency scores prediction but on TVSum, even if i try to use the tvsum_train_release.jsonl, the predictions still were identical . Or could -9.4296 have some special meaning? Looking forward to your response and thank you for your assistance!

[awkrail]

@Andzy324

Could you provide your intermediate raw predictions of saliency scores, namely accessing preds = outputs['saliency_scores'] of your experiments

I think that the weight files are open, so you can easily reproduce experiments.

Besides, have you ever paid attention to the numeric predictions of UVCOM instead of mAP metric when running experiment on TVSum dataset?

No, as you pointed out, they do not release weights on TVSum, so I reproduced their mAP scores, but actually not checked the outputs carefully on TVSum.

Using lighthouse to conduct evaluation on QVHighlight, i can get non-uniform saliency scores prediction but on TVSum, even if i try to use the tvsum_train_release.jsonl, the predictions still were identical .

In my opinion, one of the main reason is overfitting. Compared with QVHighlights (>10,000 videos), TVSum has only 50 videos. In addition, they split the videos with 10 domains, so the number of training videos is very little. This may make the model to overfit the training data, and predict the uniform scores. I will investigate this problem, but not sure now. -9.4296 itself has no meaning.
If you have a time, could you run CG-DETR or other repo that supports TVSum and report the prediction outputs here? Thanks.

[Andzy324]

@awkrail ,

Thank you for the detailed analysis and actionable suggestions! 🙏 Your insights into potential misconfiguration of domain parameters, feature alignment, and dataset limitations are helpful. Regarding numeric saliency predictions:
i will attempt to reproduce UVCOM/CG-DETR’s outputs on TVSum and share observations here recently. i will update this thread with my progress.

[Andzy324]

@awkrail
i have conducted some runs with checkpoints and features of lighthouse, the results of experiments conducted on TVSum domain BK using CG_DETR, QD_DETR, moment_DETR are as follows:

and this picture below depicts using UVCOM run on TVSum VT domain:

[awkrail]

@Andzy324
Thank you for reporting the scores. This indicates that UVCOM tends to output the uniform scores. I will investigate this issue.

[awkrail]

@Andzy324
If you have a time, could you implement the training/evaluation code on UVCOM repo and report the scores? I will try this issue on this week, but I am glad if you help it.

[Andzy324]

@awkrail
Thanks for clarifying! I’ll work on implementing/running the UVCOM training/evaluation script later this week using the checkpoints and features shared on lighthouse homepage. And i will share the scores here once completed.

[Andzy324]

Hi @awkrail ,

Thanks for following up on the evaluation for TVSum. I looked into it, but unfortunately, the UVCOM authors did not provide a script specifically for training or evaluating on the TVSum dataset, and implementing one entirely from scratch is a bit beyond my current scope. I’m sorry I can’t be more helpful on that front.

However, I did run some experiments using the provided checkpoints on both the TVSum dataset using i3d_clip features setup and the YTBHighlight dataset using clip_slowfast features setup. Below is a brief summary of the mAP metrics I observed (please see the attached table for details):
TVSum Results

Model	VT	VU	GA	MS	PK	PR	FM	BK	BT	DS	Avg.
CG-DETR (Paper)	86.9	88.8	94.8	87.7	86.7	89.6	74.8	93.3	89.2	75.9	86.8
CG-DETR	90.139	40.417(!)	96.139	42.958(!)	53.958(!)	38.514(!)	43.444(!)	93.188	58.052(!)	58.111(!)
UVCOM (Paper)	87.6	91.6	91.4	86.7	86.9	86.9	76.9	92.3	87.4	75.6	86.3
UVCOM	68.639(!)	87.917	63.354(!)	24.319(!)	52.729(!)	46.299(!)	42.972(!)	92.361(!)	86.976 (slightly diff.)	78.285 (slightly diff.)

YTBHight Results

Model	Dog	Gym.	Par.	Ska.	Ski.	Sur.	Avg.
CG-DETR (Paper)	76.3	76.1	70.0	76.0	75.1	81.9	75.9
CG-DETR	77.334	74.808	67.453	73.087	74.814	82.27	74.961
UVCOM (Paper) (UMT)	66.5	77.4	82.8	78.7	74.2	84.6	77.4
UVCOM	75.147	75.524	67.946(!)	74.649	73.904	78.921	74.3485

Observations & Questions
For TVSum, the results from the checkpoints are significantly different from those reported in the paper. In particular, some metrics show a clear drop in performance, and there is an issue with uniform predicted saliency scores in some cases (those marked with ! in TVSum result table).
On the other hand, YTBHighlight results are more consistent with the paper, showing only minor variations (the performance on parkour domain does not show uniform predictions but is relatively worse, which also aligned with the issue you have posted on UVCOM repo).
Given this discrepancy, one possible explanation could be the selection of checkpoints for TVSum evaluation.
Could it be that the training of checkpoints can be refined to select the best-performing one of each domain for TVSum?
Or could there be an issue related to this specific dataset or evaluation setup?

Thanks again for your insight!

[awkrail]

@Andzy324 Thank you for your detailed experiments. I confirmed that CG-DETR worked well on TVSum..., but in your experiments, both CG-DETR and UVCOM does not perform well on TVSum and YTBHighlight. Hmm...

Could it be that the training of checkpoints can be refined to select the best-performing one of each domain for TVSum?
Or could there be an issue related to this specific dataset or evaluation setup?

In the current situation, I am not sure. I need to investigate the difference for details, probably several difference exist between our implementation and CG-DETR/UVCOM. CG-DETR authors released the implementation, so I will check the difference for details.