[FFmpeg-devel] [PATCH] libvpx: alt reference frame / lag

[John Koleszar]

On Thu, Jun 17, 2010 at 2:10 PM, Michael Niedermayer <michaelni at gmx.at> wrote:
> On Thu, Jun 17, 2010 at 11:42:28AM -0400, John Koleszar wrote:
>> On Thu, Jun 17, 2010 at 12:24 AM, Reimar D?ffinger
>> <Reimar.Doeffinger at gmx.de> wrote:
[...]
>> Here's another example: consider a VNC type application. You could use
>> the ARF buffer to keep a low-quality version of parts of the screen
>> that are currently hidden. You might go a long time without ever
>> updating a visible part of the screen, but still do updates to the
>> hidden part of the screen. This could give a better experience when
>> dragging windows around. In this case, there would be many invisible
>> packets, and there wouldn't be a visible packet to pack them with.
>
> and a codec specifically designed for a VNC like application, that is
> one that supported really moving windows around and keeping track of
> the background would be by a big step more efficient.
>

There's a lot of interest in using VP8 for screen sharing and other
VNC type apps. Obviously this is an area that the application has a
lot of information that would be useful to pass to an encoder, and
while the current encoder isn't tuned for this application, there are
some features exposed already are interesting for these apps, and we
could do more in the future. Using VP8 as the wire protocol here is
sound, and ARFs are useful here.

>
>>
>> >> I've been through a lot of the advantages of keeping the data
>> >> separate, but it mostly boils down to staying out of the way of
>> >> applications that know what they're doing
>> >
>> > I think you are arguing that we should make things more complex for
>> > everyone for the sake of simplifying things for a use-case that
>> > currently does not exist and you don't specify well enough so we
>> > could suggest and alternative.
>>
>> I don't want to be in the business of saying what people can and can't
>> do with the codec, so I'm looking to provide flexibility. Yes, ARFs
>> are a new feature that other codecs don't have. No, I don't know all
>> the ways that people will find to use them yet.
>
> But we expect that ARFs will cause problems to some players.
> Now if you showed that they where usefull and did improve quality relative
> to B frames alone then iam sure people would not be so much opposed but
> simply arguing that they might be usefull is kinda weak to argue that
> many applicaions would have to be changed to deal with the new special
> cases.
>

They are useful, they do improve quality relative to not having them
(that blog post showed about 1dB, it's obviously material and ARF mode
(noise reduced vs b-frame-like lookahead) dependant). They're being
used today. I'm just trying to say they have wider application than
what they're being used for today.

> For example if you implemented ARFs in a codec that supported B frames
> like x264 and our h264 decoder and that led to a non trivial quality gain
> iam sure there would be very little opposition.
>
> Also, who would fix all the issues caused by seperately coded ARFs /non
> displayed frames? thats changes to the nut spec and our implementation
> changes to the libavformat core probably, maybe changes in mplayer,
> maybe other players ...
> ?
>

Obviously we have a vested interest in making sure this works well as
broadly as possible, particularly ffmpeg, mplayer, mencoder,
gstreamer, etc, and the goal is to get clean support.. if that means
changing the lavf core, we can do that. Maybe what we need to do is
create a pathological file to test what apps out there are affected..
Create an ARF every other packet or something and see if the av sync
drifts. I'm still inclined to think that as long as you're keeping
time on the output of the decoder rather than the input, this isn't a
big deal. Even the simple lavc tutorial gets this right (the decoder
not always outputting packets, it doesn't keep time) so I'm hopeful
that this isn't as big a deal as it potentially could be. In what
other cases might an lavc decoder not output a packet, maybe there's
already a good test case for this that we're not thinking of. This
behavior of avcodec_decode_video() is actually one of the things that
informed the design of not always returning a packet from libvpx.

> I just think its not nice to dump such a feature on the world because
> it might for someone be usefull and expect "the world" to figure out
> how to deal with it.
>