[FFmpeg-devel] [RFC] libavfilter-soc: direct rendering

[Vitor Sessak]

Hi, and sorry for the delay.

Stefano Sabatini wrote:
> On date Sunday 2009-06-07 16:10:38 +0200, Stefano Sabatini encoded:
>> I'll try to sketch the system as I currently understood it from the
>> discussion.
>>
>> avfilter_get_video_buffer(link, dst->min_perms)
>>
>> request should be passed up to the sink of the chain.  If the
>> destination pad of the link defines a get_video_buffer() callback,
>> then it should be used that, otherwise it will be called recursively
>> avfilter_get_video_buffer() on the first output link of the filter.
>>
>> If the filter has no output links, then the
>> avfilter_default_get_video_buffer() should be called, which will
>> allocate a video buffer using av_malloc() and friends.
>>
>> The avfilter_get_video_buffer() may pass through some filter which may
>> request to change the size of the video buffer, for example a pad
>> filter.
>>
>> Let's consider this example:
>>
>> +----------+             +------------+            +-----------+
>> |          |    link1    |            |   link2    |           |
>> |   in     +-------------+    pad     +------------+    out    |
>> | (source) |   (h1, w1)  | (filter)   |  (h2, w2)  |  (sink)   |
>> +----------+             +------------+            +-----------+
>>
>> The "in" source invokes avfilter_get_video_buffer() on link1, which
>> calls avfilter_get_video_buffer() on link2, since out doesn't define a
>> get_video_buffer() in its input pad and it has no output links, then
>> avfilter_default_get_video_buffer() is called.
>>
>> Now avfilter_get_video_buffer() allocates a buffer with size (h2, w2),
>> which takes in consideration also the padding area.
>>
>> This can be achieved defining in the pad filter a config_links in the
>> output pad, which redefines the h2 and w2 sizes using the relations:
>>
>> h2 = h1 + padtop  + padbottom;
>> w2 = w1 + padleft + padright;
>>
>> So at the end the in source will have an AVFilterPicRef with size (h2,
>> w2). Since in is supposed to write in the non padded area, it has to
>> know the information related to the padding, and so we need some
>> mechanism to propagate this information.
>>
>> Now let's consider a more complicated example:
>>
>> +--------+           +---------+          +----------+       +------+       +------+
>> |        |  link1    |         |  link2   |          |link3  |      | link4 |      |
>> | in     +-----------+   pad1  +----------+  scale   +-------+ pad2 +-------+ out  |
>> |(source)| (h1, w1)  |(filter) | (h2, w2) |  (filter)|(h3,w3)|      |(h4,w4)|      |
>> +--------+           +---------+          +----------+       +------+       +------+
>>
>> In this case the scale input pad will define get_video_buffer callback
>> which will allocate a buffer (using avfilter_default_get_buffer), and
>> its start_frame function will request a frame to the rest of the
>> chain.
>>
>> When the in source will be requested a frame by
>> avfilter_request_frame, it will request the frame to the filter chain
>> using the avfilter_get_video_buffer(), then it will update it
>> using the padding information (to change the picref->data offsets),
>> and finally will call avfilter_start_frame(), passing a picture
>> reference to the following filter.
>>
>> Padding informations (padleft, padright, padtop, padbottom) may be
>> stored in the link itself (for example when configuring the filters).
>>
>> This scheme should allow for direct rendering, since the out sink may
>> directly provide a buffer to the source filter where to directly
>> write, avoiding the initial memcpy.
>>
>> Would such a scheme fit well?
> 
> Attached patches implement this new system, try them with:
> ffplay in.avi -vfilters "pad = exp_w=in_w+30 : exp_h=in_h + 30 : color = red,
>                          pad = exp_w=in_w+30 : exp_h=in_h + 30 : color = blue,
>                          pad = exp_w=in_w+30 : exp_h=in_h + 30 : color = yellow"
> 
> (no I didn't tested them extensively).

May I suggest again that you test your patches primarily with ffmpeg.c 
(which is unchanged by your patch, probably will fail compilation)? 
ffplay cheats by doing a memcpy() for every frame, what makes it 
specially forgiving of libavfilter bugs. Besides that, I think users 
will use filters much more with ffmpeg than with ffplay.

-Vitor