No scaling for VMAF metric to match model resolution

[ataluts]

The program seems to scale video inputs only if their resolution doesn't match each other. But to get meaningful result in VMAF metric both video inputs must be scaled to the model resolution (FHD or 4k). So when the video resolution differ from the model's resolution VMAF results are wrong.

For example when I test my digitized VHS videos with resolution of 752*564 I get significantly higher VMAF value than it actually should be (99.3258 vs 97.29893 reported by ab-av1 or 97.215963 reported by ffmpeg itself when i scale input manually). These video files are compared as is but must be scaled to 1440*1080 to get correct result.

The actual ffmpeg command (from the log) is:
ffmpeg -hide_banner -nostdin -r 50 -i F:/FFMetrics_test/distorted_av1_752x564_50fps_10b.mkv -r 50 -i F:/FFMetrics_test/reference_ffv1_752x564_50fps_8b.mkv -lavfi [0:v]settb=AVTB,setpts=PTS-STARTPTS,format=yuv420p[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:log_fmt=json:log_path='C\:/Users/Alexander/AppData/Local/Temp/FFMetrics/51f1355b-b875-473d-a69e-45b7d81646c6.log':n_threads=15:pool=Mean:model=version=vmaf_v0.6.1 -f null -

but should be:
ffmpeg -hide_banner -nostdin -r 50 -i F:/FFMetrics_test/distorted_av1_752x564_50fps_10b.mkv -r 50 -i F:/FFMetrics_test/reference_ffv1_752x564_50fps_8b.mkv -lavfi [0:v]scale=1440:1080:flags=bicubic,settb=AVTB,setpts=PTS-STARTPTS,format=yuv420p[main];[1:v]scale=1440:1080:flags=bicubic,settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:log_fmt=json:log_path='C\:/Users/Alexander/AppData/Local/Temp/FFMetrics/51f1355b-b875-473d-a69e-45b7d81646c6.log':n_threads=15:pool=Mean:model=version=vmaf_v0.6.1 -f null -

Log of the program in debug mode: FFMetrics_log-debug.log

[fifonik]

You right. Probably :-)
VMAF developers mentioning that in their FAQ, but all their own examples just showing how to calculate the metric in originl resolution.
In addition, scaling "reference" introduces additional artifacts and to be honest I do not know what is better -- to have VMAF metric taking into account wrong FOW or to calculate metric where reference has these scaling artifacts.

So I will not consider this as a bug.
I will probably inmplement a checkbox (ticked off by default) that can be used to force VMAF calculation to scale both inputs to model's resolution.

[ataluts]

VMAF was designed to compare subjective quality of a video when viewed by a human at the specific distance from the screen (3 times image height). As it was done by Netflix I presume it was primarely for watching movies from streaming service on a TV. This setup assumes it is watched fullscreen with scaling to that screen if input resolution is lower, not in native resolution with black borders around. So "introducing additional artifacts by scaling" is already assumed in this metrics.

The model expects input videos to be in specific resolution (which the model itself was trained at). If inputs do not match that resolution it is equivalent to changing the viewing distance hence angular size of introduced artifacts which directly affect the perceived quality.

So, technically, you can calculate VMAF value in native video resolution but for every case you will need to come up yourself which value is "good" and which is "bad". In this case you can't use conventional metrics from the Internet (like 95+ is "good" and 97+ is "excellent") and can't compare results between different resolutions.

Checkbox to enable/disable VMAF scaling is a good solution but scaling really should be enabled by default. Because "default" user most likely will not investigate into VMAF nuances and instead will simply search which VMAF value is considered "good" and will target that in encodes. Enabling scaling ensures results correspond to the expected conventional interpretation of VMAF scores.

[ataluts]

ab-av1 approach to scaling seems quite reasonable: if resolution of inputs is close enough to the model - do not scale, if it differs signnificantly - do scale. The thresholds there are width<1728 or height< 972 for FHD model and width<3456 or height<1944 for 4k model.
Additionaly you can implement configuration of these thresholds in the settings which will define what resolutions will be scaled and to what model.

[fifonik]

In version v1.6.5 beta 1 (just uploaded) a new checkbox "VMAF | Upscale to model's resolution" is added (should be ticked automatically for new projects).
Models' width/height can be configured, see FFMetrics.conf.example.
Thresholds cannot be configured and 90% of model's width/height is used at this moment (this is the values mentioned in your previous post).
If you created FFMetrics.conf, you need to adjust it by adding {{ref-scale}} to VMAF's LAVFI option as per FFMetrics.conf.example.

[ataluts]

This works well, thanks!
But everything is simple only when both videos are in lower resolution than the model. Other cases are not so obvious:

1. When resolution of both videos is higher than in the model.
   Following VMAF logic both videos should be scaled down to match the model's resolution to get standard metric score.
2. When reference is higher resolution and distorted is lower resolution than the model.
   By the same logic reference should be scaled down and distorted scaled up.
3. Previous case but the other way around - reference lower, distorted higher.
   Reference should be scaled up and distorted scaled down. I don't think that this is realistic scenario but technically possible.

Taking into account that there are at least two models with different resolutions (FHD and 4k) there is even grayer zone in between those resolutions.

So I think better name for this option would be more general "Scale to model's resolution" instead of "Upscale to model's resolution".
I am not 100% sure how to deal with these cases but ChatGPT tells that one must always scale to the model's resolution and this seems legit.
Also the same threshold logic can be applied if resolution is higher than the model's. Like do not scale down if 10% higher than needed.

[Hlsgs]

Is there a command line option to have this unchecked when opening the program?

[fifonik]

Is there a command line option to have this unchecked when opening the program?

Actually, such option is exist:
--vmaf-upscale-to-model=0 (or 'false')
I've forgotten to mention it on main page.