[FFmpeg-devel] [PATCH] libvpx: alt reference frame / lag

[John Koleszar]

On Wed, Jun 16, 2010 at 1:15 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
> On Tue, Jun 15, 2010 at 05:09:49PM -0400, John Koleszar wrote:
>> On Tue, Jun 15, 2010 at 3:15 PM, Michael Niedermayer <michaelni at gmx.at> wrote:
>> > On Tue, Jun 15, 2010 at 12:18:29PM -0400, John Koleszar wrote:
> [...]
>> >> the decode side, it's useful for the application to be able to do work
>> >> between decoding the two packets.
>> >
>> > What actually is the advantage of these frames compared to B frames with
>> > normal reordering?
>> >
>>
>> http://webmproject.blogspot.com/2010/05/inside-webm-technology-vp8-alternate.html
>>
>> >
>> >> Stream copy shouldn't be any problem
>> >> as long as both transports don't make an assumption about the frame
>> >> being shown. You could support it with other containers with a
>> >> bitstream filter arrangement that knows how to split/combine packets.
>> >
>> > We also could finally get AAC in LATM working properly
>> > if its so easy, please send a patch, alot of people would be very happy
>> > if someone did.
>> >
>>
>> I don't know what bitstream filters in ffmpeg can do, or what it is
>> that you're implying they're missing. I was just outlining one way
>> people could put vp8 into a container that doesn't support these
>> frames.
>
> whats missing for AAC in LATM is someone with time and will to implement
> it cleanly
>
>
>>
>> >
>> >> I'm not sure this how much value there is in trying to support these
>> >> more restrictive containers/transports. It's interesting in the
>> >> academic sense, but how practical is it?
>> >
>> > in which container besides mkv does it work?
>> >
>>
>> Should be possible in nut,
>
> no, nut does not allow it.
> It would not be a big issue to make nut support it (we just need to add
> a flag to mark such frames and code to adapt the timestamp handling)
>
>
>> and there's a mapping defined for ogg.
>
> given the amount of trouble everyone had just getting anything simple in
> ogg working and given the amount of ambiguity and difference between
> spec and implementation in ogg, i doubt that this will work out smoothly
>
>
>> Might need the additional constraint of never following a KF with an
>> ARF for some cases, which we could add. Should basically work in any
>> variable frame rate container that supports either monotonically (not
>> strictly) increasing timestamps or uses a timebase of higher
>> resolution than the frame rate.
>
> it requires a container supporting frames that have a dts but not a pts
> iam not aware of such containers but i did not check any thats just from
> memory ...
> As said for nut its a matter of a new flag. for other containers outside
> our control this would not be that easy.
> What you write above is a description of a hack not of clean support.
> because these frames do not have a pts, they are not presented, without
> marking them it will mess up our timestamp handling. and i think it would
> also mess up mplayers timestamp handling, at least mplayer had its share
> of issues with h264 PAFF due to decoder input and output not matching.
> Also just think of a player that passes a frame with dts=5 to a decoder
> prior to vp8 arf the player could have expected that the decoder would
> return a decoded frame with pts=5 for this but with arf it could output
> nothing. This has its potential to break players, iam not saying it will
> iam saying it could
>

I'm all for supporting it cleanly. mkv has a flag for this, sounds
like adding one to nut would take the same approach. Just need to
propagate that flag from the encoder through to the muxer. I don't
think this would be a huge change in lavc/lavf, but feedback on that
approach is appreciated.

Consider a packing that required you put a b-frame into the same
packet as a p-frame. Every disadvantage I can think of for such a
system applies to putting ARFs into the same packet as another frame.
You'd need non-trivial queuing on the encoder, which introduces
latency. You'd need asynchronous or partial decoding on the decoder,
which may or may not have an unusual API, or have to accept additional
latency/jitter. Applications that did want to get at the frames
individually would need to duplicate the sort of parsing that's
usually done by a demuxer. This all strikes me as needlessly
complicated. A packet stream is a clean abstraction that everybody
understands, the only twist here is that not all packets are
displayed. This sort of N-in M-out behavior is common in other parts
of the filter chain, so most players/apps already have this properly
abstracted.

On the decode side, I don't think not returning a frame is a big deal.
I see what you're saying about timestamps, but consider a decoder that
chooses not to display a frame if it's known corrupt, or maybe one
that knows that it's identical to the last frame. You need some way to
signal this no-op down the filter chain, and this is a variant of that
case. Certainly a filter could conceivably decide that it wanted to
take action on that frame even though it's identical, and you'd
probably only want to do that on presented frames. (I'm just thinking
out loud here). I agree that frames that aren't to be presented are a
special case, but this works on a pretty wide variety of players, and
I can't imagine a broken player where the fix wouldn't be pretty
trivial (famous last words).

I agree that what I've described is a hack, but I don't think it's
egregious. And what we're talking about is stuffing VP8 into a
container that doesn't support it, which is a hack in and of itself.
We shouldn't need to do anything crazy to put this into mkv, and I'd
love to see native support in nut too, but both of those require
adding support for these invisible frames to the ffmpeg core.