[FFmpeg-user] should I shoot the dog?

[Devin Heitmueller]

Hi Mark,

On Tue, Sep 29, 2020 at 10:51 AM Mark Filipak (ffmpeg)
<markfilipak at bog.us> wrote:
> I imagine that format-to-format conversion is probably the most optimized code in ffmpeg. Is there a
> function library dedicated solely to format conversion? I ask so that, in what I write, I can assure
> users that the issues are known and addressed.

Most of these sorts of conversions for video are performed by the
swscale library, which is part of ffmpeg.  When ffmpeg constructs a
filter pipeline it looks at the input and output formats for each
filter in the chain, and if they don't support the same format an
instance of the vf_scale filter is automatically inserted into the
pipeline between the two filters to perform the conversion.  It's
worth noting that in some cases this can cause significant performance
problems and more than once I've found a performance bottleneck to be
that a filter performing conversion was automatically inserted without
my knowledge.

A good deal of effort has been put into optimizing swscale, but it's a
huge matrix of conversions that are possible and hence some work
better than others.  Hence performance can vary greatly depending on
the transform being done.

> For my modest purposes, a sketch of planar v. packed is probably all that's needed. I think you've
> made "planar" clear. Thank you for that. I can imagine that the structure of packed is
> multitudinous. Why is it called "packed"? How is it packed? Are the luma & chroma mixed in one
> buffer (analogous to blocks in macroblocks) or split into discrete buffers? How are they spacially
> structured? Is there any special sub structures (analogous to macroblocks in slices)? Are the sub
> structures, if any, format dependent?

The following describes the differences between packed an planar:

https://wiki.videolan.org/YUV

With packed the luma and chroma are interleaved (YCbCrYCbCr).  With
planar they are not (YYY...CbCbCb...CrCrCr).  There is also something
known as "semi-planar" where the Y is separate but CbCr is interleaved
(YYY...CbCrCbCrCbCr).

Whether the buffers themselves are contiguous is implementation
dependent.  In ffmpeg you get pointers to the individual buffers for
each plane, but there is nothing to say that under the hood the second
buffer isn't immediately after the first buffer in the underlying
memory.  Of course you should never write code that makes this
assumption, and perform operations against each plane without assuming
the data is adjacent to the buffer for another plane.

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller at ltnglobal.com