[FFmpeg-devel] [PATCH 2/7] decode: add a method for attaching lavc-internal data to frames

[wm4]

On Mon, 16 Oct 2017 20:32:03 +0800
Xiaolei Yu <dreifachstein at gmail.com> wrote:

> On 10/16/2017 07:36 PM, wm4 wrote:
> > On Mon, 16 Oct 2017 19:28:27 +0800
> > Xiaolei Yu <dreifachstein at gmail.com> wrote:
> >   
> >> On 10/03/2017 09:15 PM, wm4 wrote:  
> >>> From: Anton Khirnov <anton at khirnov.net>
> >>>
> >>> Use the AVFrame.opaque_ref field. The original user's opaque_ref is
> >>> wrapped in the lavc struct and then unwrapped before the frame is
> >>> returned to the caller.
> >>>
> >>> This new struct will be useful in the following commits.
> >>>     
> >>
> >> How about using AVFrame::buf[]?  
> > 
> > My local crash-fix hack for videotoolbox actually uses buf[] for this
> > purpose, but it won't work in general, because video data might need
> > it. Also that would have exactly the same problems as opaque_ref.
> > Actually, all approaches will have exactly the same problems as
> > opaque_ref, but michael doesn't get it.  
> 
> I think handling foreign memory with a cpu-centric api will always be kinda
> awkward. But using AVFrame::buf[] is more natural if we treat them as a
> family of cpu-unaddressable images. Also it would be great (in the future)
> to allow users to pass them around without a round trip to the main memory.

No, AVFrame.buf[] always points to CPU addressable memory. For hardware
decoding, the AVFrame.buf[0] usually points to a dummy AVBufferRef
(with either size 0, or pointing towards some other struct describing
the hardware surface).

Not sure how this relates to the current discussion. The problem with
videotoolbox is that AVHWAccel.end_frame can be run on a worker thread,
and videotoolbox needs to set the actual hardware surface in that
callback (because Apple's API returns the frame once all decoding is
done, and you can't pass a user-allocated surface to it). The worker
thread can't propagate its AVFrame changes back to the main thread
(where another AVFrame reference to the same frame is returned). Thus a
post process callback is required, and the AVFrame within libavcodec is
always invalid from a user point of view until post process is invoked.