[MPlayer-G2-dev] slices

[D Richard Felker III]

On Mon, Feb 16, 2004 at 03:25:46AM +0100, Michael Niedermayer wrote:
> Hi
> 
> On Friday 13 February 2004 00:13, D Richard Felker III wrote:
> > Uhg, one more nasty issue about slices too!!
> >
> > Consider vf_expand with osd rendering. You want to alpha-blend osd
> > onto the image during the slice rendering, so you don't have to read
> > video memory later to add osd! But here you run into a big problem.
> > Slice rendering takes place in decoding order, whereas osd contents
> > will be known in display order. It would look really stupid for the
> > clock to jump backwards for one frame, etc. How do we solve this??
> >
> > I hate slices! No, actually slices are good. I hate B-frames!!! :/
> maybe i have an idea how to fix the slice mess, but more likely i missed many 
> serious problems as iam a bit tired :), anyway, the idea is:

:)

> a pure pull system but not of frames but slices, there would be 2 different 
> types of pull requests (more if we want, but its probably not worth)
> type 1: single slice pull (only 1 slice, nothing above, nothing below)
> type 2: all slices upto slice x
> a filter needs to support just one of them, if a filter asks for a single 
> slice but the previous filter supports just 0-x slice style then thats 
> trivial, on the other hand if a filter asks for slice 0-x but the previous 
> filter only supports single slice output then a simple buffering filter needs 
> to be inserted which would then combine the slices upto that point

Read what I just said about B frames. They're the bane of video
processing! :( Here, the problem is that you want to request frames in
display-order but have them get processed (via callbacks) in decode
order, when possible. If you pull slices, this doesn't really work.

Also, most codecs (or at least libavcodec, which is all that counts :)
don't support an api for "pulling" slices. Instead the slice function
is a callback. IMO the callback style makes more sense, but from an
API perspective, it would be nice if lavc offered both.

> slice sizes ...
> very easy too, during config each filter just asks the previous one(s) what 
> slice size it can provide, slice size can be specified with a single integer 
> pair (x,y) which splits a frame into rectangular tiles which will always be 
> requested in raster scan (left->right, top->bottom) order

Definitely not possible. Think of upside-down codecs, or flip filter
(which would swap the order slices would be drawn...), or a left-right
mirror filter, or down3dright, etc.

> many filters will probably require x=width though, iam not sure if we should 
> support x!=width at all, maybe not, if we do then we also must deal with the 
> case of converting x!=width slices to x==width

This sort of thing is the exact problem I was running into. It gets
worse when you think of something like a scale filter wanting to draw
in slices, where input pixels correspond to a factional number of
output pixels! :(

> slice draw callback from the codecs
> well, arghhh
> i probably would have to change libavcodec to support normally returning after 
> each slice and resuming, dunno how easy that would be, but i dont see an 
> alternative unless we use threads but the overhead would cancel any advantage 
> slices provide

Right, very ugly... But IMO it should be supported if it can be done
without hurting performance in the old callback mode. I really hate
when libraries take a lot of cpu time without offering to return to
the calling application, since it can force you to use threads... :(

> b frames
> each frame gets a PTS and picture type and the slice pull can ask either for 
> the next picture in display order or dont care order, so if the vo and 
> filters dont care about the order then frames could be passed in coded order 
> and the vo would do the reordering
> so we have a special case again, where a filter asks for display-order but the 
> previous one provides any-order, in which case a IPB frame reorder filter 
> needs to be inserted

Have you even thought about how any of this works with buffer
management / direct rendering? IMO there are lots of problems.

> btw if anything mentioned here has already been suggested/implemented/rejected 
> then sorry, i didnt follow the discussion about the VP lately, tell me to 
> RTFwhatever

:)))))))))))))

Actually the only thing to RTF is the secret proposal/code at
http://brightrain.aerifal.cx/~dalias/vp-in-progress/

Basically, my design right now is that we "pull" frames, but "push"
slices and buffer allocation, i.e. slices and buffer allocation happen
via callback from the previous filter(s)/codec(s). This probably has
both advantages and disadvantages. One thing I like is that it allows
filters which need to see frames in display-order to still do _some_
processing in decode-order, during the slice callback. For instance
you could compute some local noise metrics during the slice callback,
then apply denoising (including temporal denoising) only where it's
needed once you get the final frame from the previous filter.

On the other hand, it makes it impossible to do decode-order
processing of images when slices aren't supported. This could be
useful with low-resoltion movies, where a whole frame fits in the
cache as one "big slice". But if filters always use slices when
possible, this situation shouldn't arise...

Rich