[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support
matthieu.bouron at gmail.com
Thu Apr 7 14:51:44 CEST 2016
On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
> On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> matthieu.bouron at gmail.com> wrote:
>> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
>> matthieu.bouron at gmail.com> wrote:
>>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
>>> The following patch add hwaccel support to the mediacodec (h264) decoder
>>> by allowing
>>> the user to render the output frames directly on a surface.
>>> In order to do so the user needs to initialize the hwaccel through the
>>> use of
>>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
>>> The later
>>> takes a reference to an android/view/Surface as parameter.
>>> If the hwaccel successfully initialize, the decoder output frames pix
>>> fmt will be
>>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
>>> the frames on the surface:
>>> AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data;
>>> av_mediacodec_release_buffer(buffer, 1);
>>> The last argument of av_mediacodec_release_buffer enable rendering of the
>>> buffer on the surface (or not if set to 0).
>>> Regarding the internal changes in the mediacodec decoder:
>>> MediaCodec.flush() discards both input and output buffers meaning that if
>>> MediaCodec.flush() is called all output buffers the user has a reference
>>> on are
>>> now invalid (and cannot be used).
>>> This behaviour does not fit well in the avcodec API.
>>> When the decoder is configured to output software buffers, there is no
>>> issue as
>>> the buffers are copied.
>>> Now when the decoder is configured to output to a surface, the user
>>> might not
>>> want to render all the frames as fast as the decoder can go and might
>>> want to
>>> control *when* the frame are rendered, so we need to make sure that the
>>> MediaCodec.flush() call is delayed until all the frames the user retains
>>> been released or rendered.
>>> Delaying the call to MediaCodec.flush() means buffering any inputs that
>>> the decoder until the user has released/renderer the frame he retains.
>>> This is a limitation of this hwaccel implementation, if the user retains
>>> frame (a), then issue a flush command to the decoder, the packets he
>>> feeds to
>>> the decoder at that point will be queued in the internal decoder packet
>>> (until he releases the frame (a)). This scenario leads to a memory usage
>>> increase to say the least.
>>> Currently there is no limitation on the size of the internal decoder
>>> queue but this is something that can be added easily. Then, if the queue
>>> full, what would be the behaviour of the decoder ? Can it block ? Or
>>> should it
>>> returns something like AVERROR(EAGAIN) ?
>>> About the other internal decoder changes I introduced:
>>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
>>> the (hwaccel) frames can be retained by the user, we need to delay the
>>> destruction of the codec until the user has released all the frames he
>>> has a
>>> reference on.
>>> The reference counter of the MediaCodecDecContext is incremented each
>>> time an
>>> (hwaccel) frame is outputted by the decoder and decremented each time a
>>> (hwaccel) frame is released.
>>> Also, when the decoder is configured to output to a surface the pts that
>>> given to the MediaCodec API are now rescaled based on the codec_timebase
>>> those timestamps values are propagated to the frames rendered on the
>>> since Android M. Not sure if it's really useful though.
>>> On the performance side:
>>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
>>> - software output + rgba conversion goes at 59~60fps
>>> - surface output + render on a surface goes at 100~110fps
>> Patch updated with the following differences:
>> * the public mediacodec api is now always built (not only when
>> mediacodec is available) (and the build when mediacodec is not available
>> has been fixed)
>> * the documentation of av_mediacodec_release_buffer has been improved a
> Patch updated with the following differences:
> MediaCodecBuffer->released type is now a volatile int (instead of a int*)
> MediaCodecContext->refcount type is now a volatile int (instead of a
More information about the ffmpeg-devel