[FFmpeg-user] Disparity in VMAF scores

[Kamaldeep Tumkur]

On Mon, 9 May 2022 at 21:49, Clayton Macleod <cherrytwist at gmail.com> wrote:

> Using the official VMAF.exe is a bit more of a pain since you have to
> convert to yuv first, but being able to examine the results file is
> sometimes helpful. 75 sounds like something may still be off if the files
> don’t appear drastically different to your eye.
>
> --
> Clayton Macleod
> If no one comes from the future to stop you from doing it, then how bad of
> a decision can it really be?
>
> > On May 9, 2022, at 1:14 AM, Kamaldeep Tumkur <kamaldeep.tumkur at gmail.com>
> wrote:
> >
> > So my question is, does libvmaf implementation do any kind of detection
> of
> > audio in the distorted asset before computing a perceptual video quality
> > score.
> >
> >
> >> On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur <
> kamaldeep.tumkur at gmail.com>
> >> wrote:
> >>
> >> @Clayton, thanks for your response.
> >>
> >> It turned out that the external encoder was adding an audio track to the
> >> encode. This was throwing the libvmaf scores off. On removing the audio
> >> track and checking the encode, the score jumped from 25 to 77.64.
> >>
> >> Now there is basis to compare the ffmpeg and the external encode.
> >>
> >> Regards
> >> Kamaldeep
> >>
> >>
> >>
> >> On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur <
> kamaldeep.tumkur at gmail.com>
> >> wrote:
> >>
> >>> Hello,
> >>>
> >>> I'm new to the computation of VMAF scores using libvmaf in ffmpeg.
> While
> >>> trying out generating scores through libvmaf, I have an issue that I
> cannot
> >>> seem to find an explanation for.
> >>>
> >>> I encoded the 'crowd_run' HQ MP4 with two different encoders. The
> >>> resulting files were compared with the same source (original).
> >>>
> >>> VMAF score of ffmpeg encoded clip:
> >>>
> >>> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028
> >>>
> >>>
> >>> Command used:
> >>>
> >>> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100
> >>> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy
> -f
> >>> mpegts pass2.ts
> >>>
> >>>
> >>> VMAF score for externally encoded clip:
> >>>
> >>> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431
> >>>
> >>>
> >>>
> >>> I just don't know how to explain this disparity in generated scores
> when
> >>> framerates and resolution of the source were retained. The second clip
> >>> doesn't seem to show such a degradation expected through the score.
> >>> Attaching both the encoded clips here for analysis.
> >>>
> >>>
> >>> Could anyone point me to why the external encoder generates an output
> >>> with low VMAF score. Anything in the frame structure?
> >>>
> >>>
> >>> ffmpeg encode:
> >>>
> >>>
> >>>
> https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing
> >>>
> >>>
> >>> external encode:
> >>>
> >>>
> >>>
> https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing
> >>>
> >>>
> >>>
> >>> Thank you.
> >>>
>
>

I've been using the crowd_run y4m source at:
https://media.xiph.org/video/derf/y4m/crowd_run_1080p50.y4m
 Yes, I'd prefer a better score too. However, I've only been able to
generate an encode with 77.936 at most today, with the external encoder and
a qvbr setting. ffmpeg-based encodes are at 90.12, which is quite in the
zone. Yes, also need to check the standalone vmaf exec, but what I've
learned is using ffmpeg filters to scale up the HLS renditions to the
source resolution and frame-rate along with the libvmaf filter is easier.
Although, please note that the distorted asset with score of 77 had the
same resolution and frame rate as the source, so no additional filtering
was performed.