[FFmpeg-devel] Enhancement layers in FFmpeg

[Niklas Haas]

On Mon, 01 Aug 2022 13:17:12 +0000 Soft Works <softworkz at hotmail.com> wrote:
> From my (rather limited) angle of view, my thoughts are these:
> 
> When decoding these kinds of sources, a user would typically not only
> want to do the processing in hardware but the decoding as well.
> 
> I think we cannot realistically expect that any of the hw decoders
> will add support for this in the near future. As we cannot modify 
> those ourselves, the only way to do such processing would be a 
> hardware filter. I think, the EL data would need to be attached 
> to frames as some kind of side data (or similar) and get uploaded 
> by the hw filter (internally) which will apply the EL data.

If both the BL and the EL are separate fully coded bitstreams, then
could we instantiate two independent HW decoder instances to decode the
respective planes?

> IMO it would be desirable when both of these things would/could be
> done in a single operation.

For Dolby Vision we have little choice in the matter. The EL application
needs to happen *after* chroma interpolation, PQ linearization, IPT
matrix application, and poly/MMR reshaping. These are currently all
on-GPU processes in the relevant video output codebases.

So for Dolby Vision that locks us into the design where we merely expose
the EL planes as part of the AVFrame and leave it to be the user's
problem (or the problem of filters like `vf_libplacebo`).

An open question (for me) is whether or not this is required for
SVC-H264, SHVC, AV1-SVC etc.

> As long as it doesn't have its own format, its own start time,
> resolution, duration, color space/transfer/primaries, etc..
> I wouldn’t say that it's a frame.

Indeed, it seems like the EL data is tied directly to the BL data for
the formats I have seen so far. So they are just like extra planes on
the AVFrame - and indeed, we could simply use extra data pointers here
(we already have room for 8).

> 
> Best regards,
> softworkz
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".