[Libav-user] avcodec_decode_video2 returns AVERROR_INVALIDDATA with HEVC streams, but other codecs work.

[Wowfunhappy@gmail.com]

Hello!

Around a decade ago, there was a project called FFusion (aka Perian) which leveraged libavcodec to give Apple's QuickTime Player support for more video codecs. The last version ever released was based on FFmpeg 0.8, and thus, it lacked support for modern formats like VP9 and HEVC.

I managed to update the code to work with FFmpeg 2.8.17. (I know this is still an older branch, but it's what I could manage and it has the decoders I want!). It can successfully play the formats that used to work (I tested XVID, FFV1, snow, and a few others), and it can play VP9 videos via the libvpx decoder. However, when I try to decode HEVC, avcodec_decode_video2 returns -1094995529, which I understand represents AVERROR_INVALIDDATA.

Furthermore, while I can play VP9 streams via libvpx, I had less success with FFmpeg's native vp9 decoder. It can decode videos which I encoded with FFMpeg, but 4K videos downloaded from Youtube look like this: https://i.ibb.co/n3QtK2R/Screen-Shot-2021-01-20-at-1-53-46-PM.png. 1080p videos are the same, but with 4 stripes instead of 8. A very small number of frames—I suspect they're keyframes—do display properly.

Because libvpx works fine, I'm not so concerned about the VP9 decoder, but it would be really nice if I could get HEVC working! Any idea what might be going on there? The most relevant portion of the code should be https://github.com/Wowfunhappy/FFusion/blob/263a1c50acdbbca9ab540d8f54799d61cf7d8928/FFusionCodec.c#L1501.

Thank you so much! This is actually my first time working with C, so there's a lot I don't understand!