[FFmpeg-devel] Video filter ideas

[Bobby Bingham]

Hi,

I'm close to trying to turn my filter system ideas into an actual API
now, and I'd like comments and suggestions on the overall architecture
before I get too far.  Nothing here's set in stone, so any comments
which might lead to a better design are very welcome. I've been looking
through libmpcodecs for ideas lately, and to be honest, I'm not sure I
quite grok the buffer management that's going on there, so
suggestions or explanations relating to that are especially welcome.

I have in mind a sort of combined push-pull architecture.  A filter
will request a frame from its input(s) when necessary.  They will then
push slices out (requesting in turn more frames from their inputs if
necessary) by calling its output's draw_slice().  For example:

decoder -> filter -> output

Output will request a frame from the filter, which will in turn
request a frame from the decoder.  The decoder will call the filter's
draw_slice() for each slice.  As the filter processes these, it will
call the output's draw_slice().

Now, the output frames do not have to correspond one-to-one to the
input frames.  For example, if the filter in the example is a decimate
filter which drops frames which are similar, it may only output one
frame for every N input.  Similarly, if it's a filter which increases
the framerate by interpolating between frames, it may output multiple
frames for a single input frame.  The only thing I think may be
necessary is that when the next filter requests output, you should
output at least one frame.

Question:  what about outputting frames when not requested (for
example, outputting two frames for a single request, or when
satisfying a request for output B gives you enough input to also
generate data for output A? I think most filters should handle it fine,
but what about a video out which must buffer it until it's time to
display?

Next up, slices.  I don't see why filters should implement the same
thing in two different ways, so I'd like to make everything into
slices.  All frame data is passed through the draw_slice() functions of
the filters.  As I'll touch on later though, it will be possible for a
filter to indicate that it can only handle slices which are the size of
a full frame.

Actually, lets get to that now.  Michael has said before that writing a
filer should not require complete knowledge of all the internals.  So
here are some ideas Rich proposed which help simplify filter writing:
context.  A filter can specify how much spatial and temporal context it
needs.  In the spatial sense, this corresponds to the minimum size of
the slices (we could use 0 to indicate no special requirements, and
-1 to indicate whole frame slices). If an input tries passing smaller
slices, the filter system will automatically combine multiple slices
together to satisfy this requirement.  Similarly for temporal context,
the filter system would buffer the number of frames required by the
filter for it.  A temporal context requirement of 0 would mean that the
filter system won't automatically buffer frames for the filter (either
the filter doesn't need it, or can handle it itself).  Also possible
would be a temporal context requirement of -1, which would indicate
that not only does the filter not require temporal context, but that
the frames don't even have to be applied in display order.  This would
allow for a filter graph which can process frames in decode order until
the point where they hit a filter or video output which requires
display order.  At that point, the filter system would automatically
reorder frames.

I can see how these context buffers could hurt performance.  In some
cases, the filter would just be buffering frames itself, so it wouldn't
be much different.  In other cases, it allows for lazier coding of
filters with lower performance.  It certainly doesn't require their use
- so if the filter itself can do better, I think that should be
encouraged.  But I also think that a filter system with some low
performing filters is worth more than a higher performing system which
nobody can figure out how to write filters for.

Buffer management.  This is something I'm still looking for ideas on.
I understand the idea of direct rendering - rendering into a buffer
provided by the next filter (or possible even further away) to reduce
memcpys.  What I'm struggling with is a good solution to the
requirements placed on each buffer.  For example, the decoder may
require that buffers storing frames used as references are not
modified.  Looking at mplayer, this seems to have been a problem there
as well.  It appears that direct rendering is disabled when decoding
h264 because, as far as I can tell, it breaks the assumptions mplayer
makes on how many frames can be used as references.  I'll think some
more about this, but some of you may have better ideas already on
handling buffers.

Brief list of things I don't think require much detail right now:

- if a filter works on frame sized slices, its output can be
automatically cut into normal sized slices again for the remainder of
the filters
- a lot of the places I've said "the filter system will automatically
do X" may end up getting actually done by an automatically inserted
filter.  For example a "slicify" filter to reduce frame sized slices,
or a "order" filter to put frames in display order. The end effect that
other filters won't have to worry about these details is the same.

Ideas that have crossed my mind, but I'm not sure how to work them in,
or if they are worth the trouble:

- handling frames that only change a little.  Suppose only a single
slice or two change in a frame.  Some filters could theoretically skip
a lot of processing if they knew that.
- only allocating memory for a slice at a time, and reusing that for
later slices in the frame.  Only have one frame-sized buffer at the end
where the final frame is assembled.

Alright.  Sorry for the slow start on all of this.  I think there's
more to cover, but I wanted to throw out there some of the ideas I
have. And again, I especially am looking for ideas for buffer
management.

--
Bobby Bingham