Struggling to encode old movie to AV1

[Tualua]

Hello! I'm trying to encode BDRemux of old movie.
Usually my parameters are

--avhw --codec av1 --output-depth 10 --quality best --vbr 3000k --qp-max 64 --qp-min 255 --max-bitrate 9000k --vbv-bufsize 3000k --fallback-rc --la-depth 100 --tune perceptual \
--gop-len 96 --vpp-edgelevel --colorrange auto --colormatrix auto --colorprim auto --transfer auto --chromaloc auto \
--max-cll copy --master-display copy --atc-sei auto \
--audio-disposition copy --audio-metadata copy --chapter-copy --sub-copy --sub-disposition copy \
--dolby-vision-profile 10.1 --dolby-vision-rpu copy \
--dhdr10-info copy --log-level info \
--ssim --psnr

But this movie is a challenge. With this parameters VMAF score is pretty bad, around 80. I tried this

	--codec av1 --avhw --icq 23 --output-depth 10 --va --quality best \
	--la-depth 80 --la-window-size 0 --la-quality auto  \
	--tune perceptual --extbrc --i-adapt --b-adapt --bframes 10 --b-pyramid --adapt-ltr --adapt-cqm --ref 8  \
	--gop-len 240 --gop-ref-dist 16 --level auto --profile auto \
	--colorrange auto --colormatrix auto --colorprim auto  --transfer auto --chromaloc auto \
	--max-cll copy --master-display copy --atc-sei auto --sao auto \
	--dhdr10-info copy --dolby-vision-profile 10.1 --dolby-vision-rpu copy \
	--video-metadata copy  \
	--audio-disposition copy --audio-metadata copy \
	--chapter-copy \
	--sub-copy --sub-disposition copy  --sub-metadata copy \
	--avsync auto --gpu-copy --output-thread -1 --task-perf-monitor  --log-level info --ssim --psnr

It gave me VMAF 90, but bitrate was HUGE - around 30000k.
Could you share some advice how to deal with it?

Sample is here https://drive.google.com/file/d/1HQPiIX1Nr4DGXnNLeYS_dJ3A8xMj8T1h/view?usp=sharing

Original Mediainfo

Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@High
HDR format                               : Dolby Vision, Version 1.0, Profile 8.1, dvhe.08.06, BL+RPU, no metadata compression, HDR10 compatible / SMPTE ST 2086, Version HDR10, HDR10 compatible
Codec ID                                 : hev1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 2 h 6 min
Bit rate                                 : 75.3 Mb/s
Maximum bit rate                         : 112 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 24.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.378
Time code of first frame                 : 00:00:00:00
Stream size                              : 66.6 GiB (89%)
Title                                    : Midnight Run by TheEqualizer
Writing library                          : HEVC Coding
Language                                 : English
Default                                  : Yes
Alternate group                          : 1
Encoded date                             : 2023-04-26 10:52:05 UTC
Tagged date                              : 2023-04-26 11:04:36 UTC
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0050 cd/m2, max: 4000 cd/m2
Maximum Content Light Level              : 1511 cd/m2
Maximum Frame-Average Light Level        : 715 cd/m2
mdhd_Duration                            : 7589625
Codec configuration box                  : hvcC+dvvC

[quamt]

@Tualua
I had a quick look, and there is some grain/noise in the video that would explain the large file size and the "low quality" with high compression.

As a starting point, try to remove the grain/noise and test with the below settings added to your original settings:

--vpp-knn radius=2,strength=0.08,lerp=0.35,th_lerp=0.85
--vpp-nlmeans sigma=0.0035,h=0.055,patch=5,search=11

Result with the filters enabled:

ssim/psnr: SSIM YUV: 0.985369 (18.347254), 0.997978 (26.942179), 0.998826 (29.302086), All: 0.989713 (19.877224), (Frames: 898)
ssim/psnr: PSNR YUV: 46.109629, 53.453066, 55.689015, Avg: 47.561990, (Frames: 898)

Without the filters:

ssim/psnr: SSIM YUV: 0.906359 (10.285358), 0.941591 (12.335184), 0.990643 (20.288625), All: 0.926279 (11.324061), (Frames: 898)
ssim/psnr: PSNR YUV: 39.089552, 42.038544, 49.122747, Avg: 40.237498, (Frames: 898)

What the metrics say

Big jump: Avg PSNR went +7.32 dB (40.24 - 47.56) and SSIM(All) 0.926 - 0.990.

Most of the gain is on luma (Y), which means the denoisers removed a lot of high-frequency variation (i.e., grain/noise). That makes frames easier to encode, and scores shoot up.

What that means visually

Metrics reward smoothness.
So this result is consistent with a cleaner but possibly “too smooth” look.

This is a starting point; adjust the filter settings to achieve the best result for you.

Quick checks (no metrics needed)

• Step through motion (face/hand) for temporal smearing.
• Look at the red patterned seat and skin pores for a waxy/painted look.
• Check dark walls for banding (smoothing can expose it after encoding).

[Tualua]

Thank you very much! I will try that

…

On Mon, Sep 29, 2025 at 8:58 PM quamt ***@***.***> wrote: *quamt* left a comment (rigaya/QSVEnc#270) <#270 (comment)> @Tualua <https://github.com/Tualua> I had a quick look, and there is some grain/noise in the video that would explain the large file size and the "low quality" with high compression. As a starting point, try to remove the grain/noise and test with the below settings added to your original settings: --vpp-knn radius=2,strength=0.08,lerp=0.35,th_lerp=0.85 --vpp-nlmeans sigma=0.0035,h=0.055,patch=5,search=11 Result with the filters enabled: ssim/psnr: SSIM YUV: 0.985369 (18.347254), 0.997978 (26.942179), 0.998826 (29.302086), All: 0.989713 (19.877224), (Frames: 898) ssim/psnr: PSNR YUV: 46.109629, 53.453066, 55.689015, Avg: 47.561990, (Frames: 898) Without the filters: ssim/psnr: SSIM YUV: 0.906359 (10.285358), 0.941591 (12.335184), 0.990643 (20.288625), All: 0.926279 (11.324061), (Frames: 898) ssim/psnr: PSNR YUV: 39.089552, 42.038544, 49.122747, Avg: 40.237498, (Frames: 898) *What the metrics say* *Big jump:* Avg PSNR went +7.32 dB (40.24 - 47.56) and SSIM(All) 0.926 - 0.990. Most of the gain is on luma (Y), which means the denoisers removed a lot of high-frequency variation (i.e., grain/noise). That makes frames easier to encode, and scores shoot up. *What that means visually* Metrics reward smoothness. So this result is consistent with a cleaner but possibly “too smooth” look. *This is a starting point; adjust the filter settings to achieve the best result for you.* *Quick checks (no metrics needed)* - Step through motion (face/hand) for temporal smearing. - Look at the red patterned seat and skin pores for a waxy/painted look. - Check dark walls for banding (smoothing can expose it after encoding). — Reply to this email directly, view it on GitHub <#270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHX2DHN4VC5JM5NJVDM2R4L3VEUHNAVCNFSM6AAAAACHWBAJX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNBWG44TOOJWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>