.Mov IPhone Video - comparing .mov to .mkv error

[Memphiisto]

Hi!

There is a problem when trying to compare an original iphone .mov file to a converted file (mkv or mp4... doesnt matter)
It only gives me the values "error" for PSNR, SSIM and VMAF...

The log states:

Stream #0:1 -> #0:1 (vorbis (native) -> pcm_s16le (native))
[Parsed_libvmaf_5 @ 00000184eaf50480] input width must match.
[Parsed_libvmaf_5 @ 00000184eaf50480] input height must match.
[Parsed_libvmaf_5 @ 00000184eaf50480] Failed to configure input pad on Parsed_libvmaf_5
[fc#0 @ 00000184e9188680] Error reinitializing filters!
[fc#0 @ 00000184e9188680] Task finished with error code: -22 (Invalid argument)
[fc#0 @ 00000184e9188680] Terminating thread with return code -22 (Invalid argument)
[vost#0:0/wrapped_avframe @ 00000184e920be80] [enc:wrapped_avframe @ 00000184e9209880] Could not open encoder before EOF
[vost#0:0/wrapped_avframe @ 00000184e920be80] Task finished with error code: -22 (Invalid argument)
[vost#0:0/wrapped_avframe @ 00000184e920be80] Terminating thread with return code -22 (Invalid argument)
[out#0/null @ 00000184e99fbec0] Nothing was written into output file, because at least one of its streams received no packets.
Conversion failed!

I have an idea what it could or even probably is going to be...
IPhone .mov files store the rotation of the video only in metadata... it doesnt actually rotate the pixels...

Meaning --> The Original is in reality 3840x2160 but rotated 90° via Metadata (for Portrait)
The converted video is probably 2160x3840 without metadata rotation.

is there anything we can do inside the code to tell it to take that into account?
Maybe that it uses the rotation in metadata and auto rotates for comparison?

thanks!
Mem

Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'C:/Users/Marcel/Desktop/FIndVideos/IMG_5569.MOV':
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
creation_time : 2020-08-06T11:19:40.000000Z
com.apple.quicktime.location.ISO6709: +47.4975+009.6275+396.698/
com.apple.quicktime.make: Apple
com.apple.quicktime.model: iPhone 7 Plus
com.apple.quicktime.software: 13.3
com.apple.quicktime.creationdate: 2020-08-06T13:19:40+0200
Duration: 00:00:02.50, start: 0.000000, bitrate: 48428 kb/s
Stream #1:00x1: Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 3840x2160, 48305 kb/s, 30 fps, 30 tbr, 600 tbn (default)
Metadata:
creation_time : 2020-08-06T11:19:40.000000Z
handler_name : Core Media Video
vendor_id : [0][0][0][0]
encoder : H.264
Side data:
displaymatrix: rotation of -90.00 degrees

[fifonik]

How ffmetrics shows both files frame sizes?
Have you tried to investigate the ffmpeg command that do the comparison? (you have not provided it)

[hashhar]

ffmpeg by default applies any rotation specified in metadata during decoding. For this reason almost always all tools should use -noautorotate before EACH input file unless the tool itself handles rotation.

I can provide both details here:

This is the command that FFMetrics ran to identify files:

PS F:\desktop_encode> ffmpeg -hide_banner -nostdin -t 00:00:02 -i F:\desktop_encode\test.mp4 -frames:v 60 -filter:v idet -f null -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'F:\desktop_encode\test.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2024-08-24T17:34:59.000000Z
    location        : +52.2306+020.9829/
    location-eng    : +52.2306+020.9829/
    com.android.version: 14
    com.android.capture.fps: 60.000000
  Duration: 00:00:16.81, start: 0.000000, bitrate: 142969 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160, 143295 kb/s, 59.20 fps, 60 tbr, 90k tbn, start 0.069100 (default)
    Metadata:
      creation_time   : 2024-08-24T17:34:59.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
    Side data:
      Display Matrix: rotation of -90.00 degrees
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 256 kb/s (default)
    Metadata:
      creation_time   : 2024-08-24T17:34:59.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
[Parsed_idet_0 @ 00000224c77dec40] Repeated Fields: Neither:     0 Top:     0 Bottom:     0
[Parsed_idet_0 @ 00000224c77dec40] Single frame detection: TFF:     0 BFF:     0 Progressive:     0 Undetermined:     0
[Parsed_idet_0 @ 00000224c77dec40] Multi frame detection: TFF:     0 BFF:     0 Progressive:     0 Undetermined:     0
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    com.android.capture.fps: 60.000000
    location        : +52.2306+020.9829/
    location-eng    : +52.2306+020.9829/
    com.android.version: 14
    encoder         : Lavf62.6.101
  Stream #0:0(eng): Video: wrapped_avframe, yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive), 2160x3840, q=2-31, 200 kb/s, 60 fps, 60 tbn (default)
    Metadata:
      encoder         : Lavc62.19.100 wrapped_avframe
      creation_time   : 2024-08-24T17:34:59.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
    Side data:
      Mastering display metadata: has_primaries:1 has_luminance:1 r(0.6800,0.3200) g(0.2650,0.6900) b(0.1500 0.0600) wp(0.3127, 0.3290) min_luminance=0.005000, max_luminance=1000.000000
      Content light level metadata: MaxCLL=1000, MaxFALL=200
  Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      encoder         : Lavc62.19.100 pcm_s16le
      creation_time   : 2024-08-24T17:34:59.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
[Parsed_idet_0 @ 00000224c7767700] Repeated Fields: Neither:    61 Top:     0 Bottom:     002.06
[Parsed_idet_0 @ 00000224c7767700] Single frame detection: TFF:     0 BFF:     0 Progressive:    61 Undetermined:     0
[Parsed_idet_0 @ 00000224c7767700] Multi frame detection: TFF:     0 BFF:     0 Progressive:    61 Undetermined:     0
[out#0/null @ 00000224c774c400] video:25KiB audio:204KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=   60 fps= 22 q=-0.0 Lsize=N/A time=00:00:01.08 bitrate=N/A speed=0.406x elapsed=0:00:02.67

Even here you can see: 3840x2160 in input but output applied the rotation from "Side data" and it's missing in Output and Output became 2160x3840.

Adding -noautorotate to the command manually fixes this (for both idet and metric commands).

[hashhar]

Alternatively everyone who encodes can fix it on their end by using -noautorotate as input option (so no rotation is performed) when encoding videos AND verify that the metadata is being carried over (-map_metadata 0 for example) (so rotation is done when next tool reads file - same as happens for input file).

OR

When encoding let rotation get applied AND reset rotation metadata so that tools don't end up reapplying it.

[fifonik]

ffmpeg by default applies any rotation specified in metadata during decoding. For this reason almost always all tools should use -noautorotate

I do not agree.
If I have a FullHD vertical footage with 90 degree rotation specified in metadata. So when played, it is displayed horizontally.
Now I'm editing it: manually rotated it in NLE, applied some filters and exported. So the distorted video is FullHD horizontal. When played it is also displayed horizontally.
If "noautorotate" implemented, two these videos will not be possible to compare as for the program the source will be seen as vertical (despite the rotation in metadata) and the distorted -- horizontal.

[fifonik]

BTW, user can create a FFMetrics.conf (see FFMetrics.conf.example) and add the "-noautorotate" option if this is what required in his situation.