[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

Matthieu Bouron matthieu.bouron at gmail.com
Fri Mar 18 17:50:39 CET 2016


From: Matthieu Bouron <matthieu.bouron at stupeflix.com>

---

Hello,

The following patch add hwaccel support to the mediacodec (h264) decoder by allowing
the user to render the output frames directly on a surface.

In order to do so the user needs to initialize the hwaccel through the use of
av_mediacodec_alloc_context and av_mediacodec_default_init functions. The later
takes a reference to an android/view/Surface as parameter.

If the hwaccel successfully initialize, the decoder output frames pix fmt will be
AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to render
the frames on the surface:

    AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
    av_mediacodec_release_buffer(buffer, 1);

The last argument of av_mediacodec_release_buffer enable rendering of the
buffer on the surface (or not if set to 0).

Regarding the internal changes in the mediacodec decoder:

MediaCodec.flush() discards both input and output buffers meaning that if
MediaCodec.flush() is called all output buffers the user has a reference on are
now invalid (and cannot be used).
This behaviour does not fit well in the avcodec API.

When the decoder is configured to output software buffers, there is no issue as
the buffers are copied.

Now when the decoder is configured to output to a surface, the user might not
want to render all the frames as fast as the decoder can go and might want to
control *when* the frame are rendered, so we need to make sure that the
MediaCodec.flush() call is delayed until all the frames the user retains has
been released or rendered.

Delaying the call to MediaCodec.flush() means buffering any inputs that come
the decoder until the user has released/renderer the frame he retains.

This is a limitation of this hwaccel implementation, if the user retains a
frame (a), then issue a flush command to the decoder, the packets he feeds to
the decoder at that point will be queued in the internal decoder packet queue
(until he releases the frame (a)). This scenario leads to a memory usage
increase to say the least.

Currently there is no limitation on the size of the internal decoder packet
queue but this is something that can be added easily. Then, if the queue is
full, what would be the behaviour of the decoder ? Can it block ? Or should it
returns something like AVERROR(EAGAIN) ?

About the other internal decoder changes I introduced:

The MediaCodecDecContext is now refcounted (using the lavu/atomic api) since
the (hwaccel) frames can be retained by the user, we need to delay the
destruction of the codec until the user has released all the frames he has a
reference on.
The reference counter of the MediaCodecDecContext is incremented each time an
(hwaccel) frame is outputted by the decoder and decremented each time a
(hwaccel) frame is released.

Also, when the decoder is configured to output to a surface the pts that are
given to the MediaCodec API are now rescaled based on the codec_timebase as
those timestamps values are propagated to the frames rendered on the surface
since Android M. Not sure if it's really useful though.

On the performance side:

On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
  - software output + rgba conversion goes at 59~60fps
  - surface output + render on a surface goes at 100~110fps

Matthieu

---
 configure                       |   1 +
 libavcodec/Makefile             |   6 +-
 libavcodec/allcodecs.c          |   1 +
 libavcodec/mediacodec.c         | 125 ++++++++++++++++++
 libavcodec/mediacodec.h         |  85 +++++++++++++
 libavcodec/mediacodec_surface.c |  66 ++++++++++
 libavcodec/mediacodec_surface.h |  31 +++++
 libavcodec/mediacodec_wrapper.c |   5 +-
 libavcodec/mediacodecdec.c      | 272 +++++++++++++++++++++++++++++++++-------
 libavcodec/mediacodecdec.h      |  17 +++
 libavcodec/mediacodecdec_h264.c |  23 ++++
 libavutil/pixdesc.c             |   4 +
 libavutil/pixfmt.h              |   2 +
 13 files changed, 586 insertions(+), 52 deletions(-)
 create mode 100644 libavcodec/mediacodec.c
 create mode 100644 libavcodec/mediacodec.h
 create mode 100644 libavcodec/mediacodec_surface.c
 create mode 100644 libavcodec/mediacodec_surface.h

diff --git a/configure b/configure
index e5de306..4d66673 100755
--- a/configure
+++ b/configure
@@ -2530,6 +2530,7 @@ h264_d3d11va_hwaccel_select="h264_decoder"
 h264_dxva2_hwaccel_deps="dxva2"
 h264_dxva2_hwaccel_select="h264_decoder"
 h264_mediacodec_decoder_deps="mediacodec"
+h264_mediacodec_hwaccel_deps="mediacodec"
 h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
 h264_mmal_decoder_deps="mmal"
 h264_mmal_decoder_select="mmal"
diff --git a/libavcodec/Makefile b/libavcodec/Makefile
index 6bb1af1..a3dad7e 100644
--- a/libavcodec/Makefile
+++ b/libavcodec/Makefile
@@ -10,6 +10,7 @@ HEADERS = avcodec.h                                                     \
           dirac.h                                                       \
           dxva2.h                                                       \
           jni.h                                                         \
+          mediacodec.h                                                  \
           qsv.h                                                         \
           vaapi.h                                                       \
           vda.h                                                         \
@@ -91,7 +92,7 @@ OBJS-$(CONFIG_LSP)                     += lsp.o
 OBJS-$(CONFIG_LZF)                     += lzf.o
 OBJS-$(CONFIG_MDCT)                    += mdct_fixed.o mdct_float.o mdct_fixed_32.o
 OBJS-$(CONFIG_ME_CMP)                  += me_cmp.o
-OBJS-$(CONFIG_MEDIACODEC)              += mediacodecdec.o mediacodec_wrapper.o mediacodec_sw_buffer.o
+OBJS-$(CONFIG_MEDIACODEC)              += mediacodecdec.o mediacodec_surface.o mediacodec_wrapper.o mediacodec_sw_buffer.o
 OBJS-$(CONFIG_MPEG_ER)                 += mpeg_er.o
 OBJS-$(CONFIG_MPEGAUDIO)               += mpegaudio.o mpegaudiodata.o   \
                                           mpegaudiodecheader.o
@@ -734,6 +735,7 @@ OBJS-$(CONFIG_H263_VAAPI_HWACCEL)         += vaapi_mpeg4.o
 OBJS-$(CONFIG_H263_VIDEOTOOLBOX_HWACCEL)  += videotoolbox.o
 OBJS-$(CONFIG_H264_D3D11VA_HWACCEL)       += dxva2_h264.o
 OBJS-$(CONFIG_H264_DXVA2_HWACCEL)         += dxva2_h264.o
+OBJS-$(CONFIG_H264_MEDIACODEC_HWACCEL)    += mediacodec.o
 OBJS-$(CONFIG_H264_VAAPI_HWACCEL)         += vaapi_h264.o
 OBJS-$(CONFIG_H264_VDA_HWACCEL)           += vda_h264.o
 OBJS-$(CONFIG_H264_VDPAU_HWACCEL)         += vdpau_h264.o
@@ -947,7 +949,7 @@ SKIPHEADERS-$(CONFIG_LIBSCHROEDINGER)  += libschroedinger.h
 SKIPHEADERS-$(CONFIG_LIBUTVIDEO)       += libutvideo.h
 SKIPHEADERS-$(CONFIG_LIBVPX)           += libvpx.h
 SKIPHEADERS-$(CONFIG_LIBWEBP_ENCODER)  += libwebpenc_common.h
-SKIPHEADERS-$(CONFIG_MEDIACODEC)       += mediacodecdec.h mediacodec_wrapper.h mediacodec_sw_buffer.h
+SKIPHEADERS-$(CONFIG_MEDIACODEC)       += mediacodecdec.h mediacodec_surface.h mediacodec_wrapper.h mediacodec_sw_buffer.h
 SKIPHEADERS-$(CONFIG_QSV)              += qsv.h qsv_internal.h
 SKIPHEADERS-$(CONFIG_QSVDEC)           += qsvdec.h
 SKIPHEADERS-$(CONFIG_QSVENC)           += qsvenc.h
diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
index 2a25d66..79ce855 100644
--- a/libavcodec/allcodecs.c
+++ b/libavcodec/allcodecs.c
@@ -78,6 +78,7 @@ void avcodec_register_all(void)
     REGISTER_HWACCEL(H263_VIDEOTOOLBOX, h263_videotoolbox);
     REGISTER_HWACCEL(H264_D3D11VA,      h264_d3d11va);
     REGISTER_HWACCEL(H264_DXVA2,        h264_dxva2);
+    REGISTER_HWACCEL(H264_MEDIACODEC,   h264_mediacodec);
     REGISTER_HWACCEL(H264_MMAL,         h264_mmal);
     REGISTER_HWACCEL(H264_QSV,          h264_qsv);
     REGISTER_HWACCEL(H264_VAAPI,        h264_vaapi);
diff --git a/libavcodec/mediacodec.c b/libavcodec/mediacodec.c
new file mode 100644
index 0000000..51dd37c
--- /dev/null
+++ b/libavcodec/mediacodec.c
@@ -0,0 +1,125 @@
+/*
+ * Android MediaCodec public API functions
+ *
+ * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include "config.h"
+
+#if CONFIG_H264_MEDIACODEC_HWACCEL
+
+#include <jni.h>
+
+#include "libavcodec/avcodec.h"
+#include "libavutil/atomic.h"
+#include "libavutil/mem.h"
+
+#include "ffjni.h"
+#include "mediacodec.h"
+#include "mediacodecdec.h"
+
+AVMediaCodecContext *av_mediacodec_alloc_context(void)
+{
+    return av_mallocz(sizeof(AVMediaCodecContext));
+}
+
+int av_mediacodec_default_init(AVCodecContext *avctx, AVMediaCodecContext *ctx, void *surface)
+{
+    int ret = 0;
+    JNIEnv *env = NULL;
+    int attached = 0;
+
+    env = ff_jni_attach_env(&attached, avctx);
+    if (!env) {
+        return AVERROR_EXTERNAL;
+    }
+
+    ctx->surface = (*env)->NewGlobalRef(env, surface);
+    if (ctx->surface) {
+        avctx->hwaccel_context = ctx;
+    } else {
+        av_log(avctx, AV_LOG_ERROR, "Could not create new global reference\n");
+        ret = AVERROR_EXTERNAL;
+    }
+
+    if (attached) {
+        ff_jni_detach_env(avctx);
+    }
+
+    return ret;
+}
+
+void av_mediacodec_default_free(AVCodecContext *avctx)
+{
+    JNIEnv *env = NULL;
+    int attached = 0;
+
+    AVMediaCodecContext *ctx = avctx->hwaccel_context;
+
+    env = ff_jni_attach_env(&attached, avctx);
+    if (!env) {
+        return;
+    }
+
+    if (ctx->surface) {
+        (*env)->DeleteGlobalRef(env, ctx->surface);
+        ctx->surface = NULL;
+    }
+
+    if (attached) {
+        ff_jni_detach_env(avctx);
+    }
+
+    av_freep(&avctx->hwaccel_context);
+}
+
+int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render)
+{
+    MediaCodecDecContext *ctx = buffer->ctx;
+    int released = avpriv_atomic_int_add_and_fetch(buffer->released, 1);
+
+    if (released == 1) {
+        return ff_AMediaCodec_releaseOutputBuffer(ctx->codec, buffer->index, render);
+    }
+
+    return 0;
+}
+
+#else
+
+AVMediaCodecContext *av_mediacodec_alloc_context(void)
+{
+    return NULL;
+}
+
+int av_mediacodec_default_init(AVCodecContext *avctx, AVMediaCodecContext *ctx, void *surface)
+{
+    return 0;
+}
+
+void av_mediacodec_default_free(AVCodecContext *avctx)
+{
+}
+
+int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render)
+{
+    return 0;
+}
+
+#endif
diff --git a/libavcodec/mediacodec.h b/libavcodec/mediacodec.h
new file mode 100644
index 0000000..f303c63
--- /dev/null
+++ b/libavcodec/mediacodec.h
@@ -0,0 +1,85 @@
+/*
+ * Android MediaCodec public API
+ *
+ * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef AVCODEC_MEDIACODEC_H
+#define AVCODEC_MEDIACODEC_H
+
+#include "libavcodec/avcodec.h"
+
+/**
+ * This structure holds a reference to a android/view/Surface object that will
+ * be used as output by the decoder.
+ *
+ */
+typedef struct AVMediaCodecContext {
+
+    /**
+     * android/view/Surface object reference.
+     */
+    void *surface;
+
+} AVMediaCodecContext;
+
+/**
+ * Allocate and initialize a MediaCodec context.
+ *
+ * When decoding with MediaCodec is finished, the caller must free the
+ * MediaCodec context with av_mediacodec_default_free.
+ *
+ * @return a pointer to a newly allocated AVMediaCodecContext on success, NULL otherwise
+ */
+AVMediaCodecContext *av_mediacodec_alloc_context(void);
+
+/**
+ * Convenience function that sets up the MediaCodec context.
+ *
+ * @param avctx codec context
+ * @param ctx MediaCodec context to initialize
+ * @param surface reference to an android/view/Surface
+ * @return 0 on success, < 0 otherwise
+ */
+int av_mediacodec_default_init(AVCodecContext *avctx, AVMediaCodecContext *ctx, void *surface);
+
+/**
+ * This function must be called to free the MediaCodec context initialized with
+ * av_mediacodec_default_init().
+ *
+ * @param avctx codec context
+ */
+void av_mediacodec_default_free(AVCodecContext *avctx);
+
+/**
+ * Opaque structure representing a MediaCodec buffer to render.
+ */
+typedef struct MediaCodecBuffer AVMediaCodecBuffer;
+
+/**
+ * Release a MediaCodec buffer and render it onto the surface the decoder is
+ * associated with.
+ *
+ * @param buffer the buffer to render
+ * @param render 1 to render the buffer onto the surface or 0 to discard the buffer
+ * @return 0 on success, < 0 otherwise
+ */
+int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render);
+
+#endif /* AVCODEC_MEDIACODEC_H */
diff --git a/libavcodec/mediacodec_surface.c b/libavcodec/mediacodec_surface.c
new file mode 100644
index 0000000..903ebe4
--- /dev/null
+++ b/libavcodec/mediacodec_surface.c
@@ -0,0 +1,66 @@
+/*
+ * Android MediaCodec Surface functions
+ *
+ * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <jni.h>
+
+#include "ffjni.h"
+#include "mediacodec_surface.h"
+
+void *ff_mediacodec_surface_ref(void *surface, void *log_ctx)
+{
+    int attached = 0;
+    JNIEnv *env = NULL;
+
+    void *reference = NULL;
+
+    env = ff_jni_attach_env(&attached, log_ctx);
+    if (!env) {
+        return NULL;
+    }
+
+    reference = (*env)->NewGlobalRef(env, surface);
+
+    if (attached) {
+        ff_jni_detach_env(log_ctx);
+    }
+
+    return reference;
+}
+
+int ff_mediacodec_surface_unref(void *surface, void *log_ctx)
+{
+    int attached = 0;
+    JNIEnv *env = NULL;
+
+    env = ff_jni_attach_env(&attached, log_ctx);
+    if (!env) {
+        return AVERROR_EXTERNAL;
+    }
+
+    (*env)->DeleteGlobalRef(env, surface);
+
+    if (attached) {
+        ff_jni_detach_env(log_ctx);
+    }
+
+    return 0;
+}
diff --git a/libavcodec/mediacodec_surface.h b/libavcodec/mediacodec_surface.h
new file mode 100644
index 0000000..0178b8a
--- /dev/null
+++ b/libavcodec/mediacodec_surface.h
@@ -0,0 +1,31 @@
+/*
+ * Android MediaCodec Surface functions
+ *
+ * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef AVCODEC_MEDIACODEC_SURFACE_H
+#define AVCODEC_MEDIACODEC_SURFACE_H
+
+#include "libavcodec/avcodec.h"
+
+void *ff_mediacodec_surface_ref(void *surface, void *log_ctx);
+int ff_mediacodec_surface_unref(void *surface, void *log_ctx);
+
+#endif /* AVCODEC_MEDIACODEC_SURFACE_H */
diff --git a/libavcodec/mediacodec_wrapper.c b/libavcodec/mediacodec_wrapper.c
index 6b3f905..621e40b 100644
--- a/libavcodec/mediacodec_wrapper.c
+++ b/libavcodec/mediacodec_wrapper.c
@@ -1306,12 +1306,9 @@ int ff_AMediaCodec_configure(FFAMediaCodec* codec, const FFAMediaFormat* format,
     int attached = 0;
     JNIEnv *env = NULL;
 
-    /* TODO: implement surface handling */
-    av_assert0(surface == NULL);
-
     JNI_ATTACH_ENV_OR_RETURN(env, &attached, codec, AVERROR_EXTERNAL);
 
-    (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, NULL, NULL, flags);
+    (*env)->CallVoidMethod(env, codec->object, codec->jfields.configure_id, format->object, surface, NULL, flags);
     if (ff_jni_exception_check(env, 1, codec) < 0) {
         ret = AVERROR_EXTERNAL;
         goto fail;
diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
index d385651..1e4d9dd 100644
--- a/libavcodec/mediacodecdec.c
+++ b/libavcodec/mediacodecdec.c
@@ -23,6 +23,7 @@
 #include <string.h>
 #include <sys/types.h>
 
+#include "libavutil/atomic.h"
 #include "libavutil/common.h"
 #include "libavutil/mem.h"
 #include "libavutil/log.h"
@@ -33,6 +34,8 @@
 #include "avcodec.h"
 #include "internal.h"
 
+#include "mediacodec.h"
+#include "mediacodec_surface.h"
 #include "mediacodec_sw_buffer.h"
 #include "mediacodec_wrapper.h"
 #include "mediacodecdec.h"
@@ -118,6 +121,10 @@ static enum AVPixelFormat mcdec_map_color_format(AVCodecContext *avctx,
     int i;
     enum AVPixelFormat ret = AV_PIX_FMT_NONE;
 
+    if (s->surface) {
+        return AV_PIX_FMT_MEDIACODEC;
+    }
+
     if (!strcmp(s->codec_name, "OMX.k3.video.decoder.avc") && color_format == COLOR_FormatYCbYCr) {
         s->color_format = color_format = COLOR_TI_FormatYUV420PackedSemiPlanar;
     }
@@ -134,7 +141,113 @@ static enum AVPixelFormat mcdec_map_color_format(AVCodecContext *avctx,
     return ret;
 }
 
-static int mediacodec_wrap_buffer(AVCodecContext *avctx,
+static void ff_mediacodec_dec_ref(MediaCodecDecContext *s)
+{
+    avpriv_atomic_int_add_and_fetch(s->refcount, 1);
+}
+
+static void ff_mediacodec_dec_unref(MediaCodecDecContext *s)
+{
+    if (!avpriv_atomic_int_add_and_fetch(s->refcount, -1)) {
+        if (s->codec) {
+            ff_AMediaCodec_delete(s->codec);
+            s->codec = NULL;
+        }
+
+        if (s->format) {
+            ff_AMediaFormat_delete(s->format);
+            s->format = NULL;
+        }
+
+        if (s->surface) {
+            ff_mediacodec_surface_unref(s->surface, NULL);
+            s->surface = NULL;
+        }
+
+        av_freep(&s->codec_name);
+        av_freep(&s->refcount);
+    }
+}
+
+static void mediacodec_buffer_release(void *opaque, uint8_t *data)
+{
+    AVMediaCodecBuffer *buffer = opaque;
+    MediaCodecDecContext *ctx = buffer->ctx;
+    int released = avpriv_atomic_int_get(buffer->released);
+
+    if (!released) {
+        ff_AMediaCodec_releaseOutputBuffer(ctx->codec, buffer->index, 0);
+    }
+
+    ff_mediacodec_dec_unref(ctx);
+    av_freep(&buffer->released);
+    av_freep(&buffer);
+}
+
+static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx,
+                                  MediaCodecDecContext *s,
+                                  ssize_t index,
+                                  FFAMediaCodecBufferInfo *info,
+                                  AVFrame *frame)
+{
+    int ret = 0;
+    AVMediaCodecBuffer *buffer = NULL;
+
+    frame->buf[0] = NULL;
+    frame->width = avctx->width;
+    frame->height = avctx->height;
+    frame->format = avctx->pix_fmt;
+    frame->pkt_pts = av_rescale_q(info->presentationTimeUs,
+                                  av_make_q(1, 1000000),
+                                  avctx->pkt_timebase);
+
+    buffer = av_mallocz(sizeof(AVMediaCodecBuffer));
+    if (!buffer) {
+        ret = AVERROR(ENOMEM);
+        goto fail;
+    }
+
+    buffer->released = av_mallocz(sizeof(*buffer->released));
+    if (!buffer->released) {
+        ret = AVERROR(ENOMEM);
+        goto fail;
+    }
+
+    frame->buf[0] = av_buffer_create(NULL,
+                                     0,
+                                     mediacodec_buffer_release,
+                                     buffer,
+                                     AV_BUFFER_FLAG_READONLY);
+
+    if (!frame->buf[0]) {
+        ret = AVERROR(ENOMEM);
+        goto fail;
+
+    }
+
+    buffer->ctx = s;
+    ff_mediacodec_dec_ref(s);
+
+    buffer->index = index;
+    buffer->pts = info->presentationTimeUs;
+
+    frame->data[3] = (uint8_t *)buffer;
+
+    return 0;
+fail:
+    if (buffer) {
+        av_free(buffer->released);
+        av_free(buffer);
+    }
+
+    av_buffer_unref(&frame->buf[0]);
+
+    ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
+
+    return ret;
+}
+
+static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
                                   MediaCodecDecContext *s,
                                   uint8_t *data,
                                   size_t size,
@@ -307,14 +420,61 @@ static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecConte
     return ff_set_dimensions(avctx, width, height);
 }
 
+
+static int mediacodec_dec_flush_codec(AVCodecContext *avctx, MediaCodecDecContext *s)
+{
+    FFAMediaCodec *codec = s->codec;
+    int status;
+
+    s->queued_buffer_nb = 0;
+    s->dequeued_buffer_nb = 0;
+
+    s->draining = 0;
+    s->flushing = 0;
+
+    status = ff_AMediaCodec_flush(codec);
+    if (status < 0) {
+        av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p", codec);
+        return AVERROR_EXTERNAL;
+    }
+
+    s->first_buffer = 0;
+    s->first_buffer_at = av_gettime();
+
+    return 0;
+}
+
 int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
                            const char *mime, FFAMediaFormat *format)
 {
     int ret = 0;
     int status;
 
+    enum AVPixelFormat pix_fmt;
+    enum AVPixelFormat pix_fmts[3] = {
+        AV_PIX_FMT_MEDIACODEC,
+        AV_PIX_FMT_NONE,
+    };
+
     s->first_buffer_at = av_gettime();
 
+    s->refcount = av_mallocz(sizeof(*s->refcount));
+    if (!s->refcount) {
+        av_log(avctx, AV_LOG_ERROR, "Failed to init decoder reference counter\n");
+        goto fail;
+    }
+    *s->refcount = 1;
+
+    pix_fmt = ff_get_format(avctx, pix_fmts);
+    if (pix_fmt == AV_PIX_FMT_MEDIACODEC) {
+        AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
+
+        if (user_ctx && user_ctx->surface) {
+            s->surface = ff_mediacodec_surface_ref(user_ctx->surface, avctx);
+            av_log(avctx, AV_LOG_INFO, "Using surface %p\n", s->surface);
+        }
+    }
+
     s->codec_name = ff_AMediaCodecList_getCodecNameByType(mime, avctx->width, avctx->height, avctx);
     if (!s->codec_name) {
         ret = AVERROR_EXTERNAL;
@@ -329,7 +489,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext *s,
         goto fail;
     }
 
-    status = ff_AMediaCodec_configure(s->codec, format, NULL, NULL, 0);
+    status = ff_AMediaCodec_configure(s->codec, format, s->surface, NULL, 0);
     if (status < 0) {
         char *desc = ff_AMediaFormat_toString(format);
         av_log(avctx, AV_LOG_ERROR,
@@ -377,7 +537,7 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
 {
     int ret;
     int offset = 0;
-    int need_flushing = 0;
+    int need_draining = 0;
     uint8_t *data;
     ssize_t index;
     size_t size;
@@ -389,15 +549,21 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
     int64_t input_dequeue_timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
     int64_t output_dequeue_timeout_us = OUTPUT_DEQUEUE_TIMEOUT_US;
 
+    if (s->flushing) {
+        av_log(avctx, AV_LOG_ERROR, "Decoder is flushing and cannot accept new buffer "
+                                    "until all output buffers have been released\n");
+        return AVERROR_EXTERNAL;
+    }
+
     if (pkt->size == 0) {
-        need_flushing = 1;
+        need_draining = 1;
     }
 
-    if (s->flushing && need_flushing && s->queued_buffer_nb <= 0) {
+    if (s->draining && need_draining && s->queued_buffer_nb <= 0) {
         return 0;
     }
 
-    while (offset < pkt->size || (need_flushing && !s->flushing)) {
+    while (offset < pkt->size || (need_draining && !s->draining)) {
         int size;
 
         index = ff_AMediaCodec_dequeueInputBuffer(codec, input_dequeue_timeout_us);
@@ -416,26 +582,37 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
             return AVERROR_EXTERNAL;
         }
 
-        if (need_flushing) {
+        if (need_draining) {
+            int64_t pts = pkt->pts;
             uint32_t flags = ff_AMediaCodec_getBufferFlagEndOfStream(codec);
 
+            if (s->surface) {
+                pts = av_rescale_q(pts, avctx->pkt_timebase, av_make_q(1, 1000000));
+            }
+
             av_log(avctx, AV_LOG_DEBUG, "Sending End Of Stream signal\n");
 
-            status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, 0, pkt->pts, flags);
+            status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, 0, pts, flags);
             if (status < 0) {
                 av_log(avctx, AV_LOG_ERROR, "Failed to queue input empty buffer (status = %d)\n", status);
                 return AVERROR_EXTERNAL;
             }
 
-            s->flushing = 1;
+            s->draining = 1;
             break;
         } else {
+            int64_t pts = pkt->pts;
+
             size = FFMIN(pkt->size - offset, size);
 
             memcpy(data, pkt->data + offset, size);
             offset += size;
 
-            status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, size, pkt->pts, 0);
+            if (s->surface) {
+                pts = av_rescale_q(pts, avctx->pkt_timebase, av_make_q(1, 1000000));
+            }
+
+            status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, size, pts, 0);
             if (status < 0) {
                 av_log(avctx, AV_LOG_ERROR, "Failed to queue input buffer (status = %d)\n", status);
                 return AVERROR_EXTERNAL;
@@ -447,8 +624,8 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
         }
     }
 
-    if (s->flushing) {
-        /* If the codec is flushing, block for a fair amount of time to
+    if (s->draining) {
+        /* If the codec is draining, block for a fair amount of time to
         * ensure we got a frame */
         output_dequeue_timeout_us = OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US;
     } else if (s->dequeued_buffer_nb == 0) {
@@ -471,15 +648,22 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
                 " flags=%" PRIu32 "\n", index, info.offset, info.size,
                 info.presentationTimeUs, info.flags);
 
-        data = ff_AMediaCodec_getOutputBuffer(codec, index, &size);
-        if (!data) {
-            av_log(avctx, AV_LOG_ERROR, "Failed to get output buffer\n");
-            return AVERROR_EXTERNAL;
-        }
+        if (s->surface) {
+            if ((ret = mediacodec_wrap_hw_buffer(avctx, s, index, &info, frame)) < 0) {
+                av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec buffer\n");
+                return ret;
+            }
+        } else {
+            data = ff_AMediaCodec_getOutputBuffer(codec, index, &size);
+            if (!data) {
+                av_log(avctx, AV_LOG_ERROR, "Failed to get output buffer\n");
+                return AVERROR_EXTERNAL;
+            }
 
-        if ((ret = mediacodec_wrap_buffer(avctx, s, data, size, index, &info, frame)) < 0) {
-            av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec buffer\n");
-            return ret;
+            if ((ret = mediacodec_wrap_sw_buffer(avctx, s, data, size, index, &info, frame)) < 0) {
+                av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec buffer\n");
+                return ret;
+            }
         }
 
         *got_frame = 1;
@@ -516,9 +700,9 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
     } else if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
         ff_AMediaCodec_cleanOutputBuffers(codec);
     } else if (ff_AMediaCodec_infoTryAgainLater(codec, index)) {
-        if (s->flushing) {
+        if (s->draining) {
             av_log(avctx, AV_LOG_ERROR, "Failed to dequeue output buffer within %" PRIi64 "ms "
-                                        "while flushing remaining frames, output will probably lack last %d frames\n",
+                                        "while draining remaining frames, output will probably lack last %d frames\n",
                                         output_dequeue_timeout_us / 1000, s->queued_buffer_nb);
         } else {
             av_log(avctx, AV_LOG_DEBUG, "No output buffer available, try again later\n");
@@ -533,39 +717,35 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s,
 
 int ff_mediacodec_dec_flush(AVCodecContext *avctx, MediaCodecDecContext *s)
 {
-    FFAMediaCodec *codec = s->codec;
-    int status;
-
-    s->queued_buffer_nb = 0;
-    s->dequeued_buffer_nb = 0;
+    if (!s->surface || avpriv_atomic_int_get(s->refcount) == 1) {
+        int ret;
 
-    s->flushing = 0;
+        if ((ret = mediacodec_dec_flush_codec(avctx, s)) < 0) {
+            return ret;
+        }
 
-    status = ff_AMediaCodec_flush(codec);
-    if (status < 0) {
-        av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p", codec);
-        return AVERROR_EXTERNAL;
+        return 1;
     }
 
-    s->first_buffer = 0;
-    s->first_buffer_at = av_gettime();
-
+    s->flushing = 1;
     return 0;
 }
 
 int ff_mediacodec_dec_close(AVCodecContext *avctx, MediaCodecDecContext *s)
 {
-    if (s->codec) {
-        ff_AMediaCodec_delete(s->codec);
-        s->codec = NULL;
-    }
-
-    if (s->format) {
-        ff_AMediaFormat_delete(s->format);
-        s->format = NULL;
-    }
-
-    av_freep(&s->codec_name);
+    ff_mediacodec_dec_unref(s);
 
     return 0;
 }
+
+int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx, MediaCodecDecContext *s)
+{
+    return s->flushing;
+}
+
+AVHWAccel ff_h264_mediacodec_hwaccel = {
+    .name    = "mediacodec",
+    .type    = AVMEDIA_TYPE_VIDEO,
+    .id      = AV_CODEC_ID_H264,
+    .pix_fmt = AV_PIX_FMT_MEDIACODEC,
+};
diff --git a/libavcodec/mediacodecdec.h b/libavcodec/mediacodecdec.h
index 36fdbf5..dae3d67 100644
--- a/libavcodec/mediacodecdec.h
+++ b/libavcodec/mediacodecdec.h
@@ -34,12 +34,17 @@
 
 typedef struct MediaCodecDecContext {
 
+    int *refcount;
+
     char *codec_name;
 
     FFAMediaCodec *codec;
     FFAMediaFormat *format;
 
+    void *surface;
+
     int started;
+    int draining;
     int flushing;
 
     int width;
@@ -79,4 +84,16 @@ int ff_mediacodec_dec_flush(AVCodecContext *avctx,
 int ff_mediacodec_dec_close(AVCodecContext *avctx,
                             MediaCodecDecContext *s);
 
+int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx,
+                                  MediaCodecDecContext *s);
+
+typedef struct MediaCodecBuffer {
+
+    MediaCodecDecContext *ctx;
+    ssize_t index;
+    int64_t pts;
+    int *released;
+
+} MediaCodecBuffer;
+
 #endif /* AVCODEC_MEDIACODECDEC_H */
diff --git a/libavcodec/mediacodecdec_h264.c b/libavcodec/mediacodecdec_h264.c
index 2d1d525..4b74fb1 100644
--- a/libavcodec/mediacodecdec_h264.c
+++ b/libavcodec/mediacodecdec_h264.c
@@ -261,6 +261,29 @@ static int mediacodec_decode_frame(AVCodecContext *avctx, void *data,
         av_fifo_generic_write(s->fifo, &input_ref, sizeof(input_ref), NULL);
     }
 
+    /*
+     * MediaCodec.flush() discards both input and output buffers, thus we
+     * need to delay the call to this function until the user has released or
+     * renderered the frames he retains.
+     *
+     * After we have buffered an input packet, check if the codec is in the
+     * flushing state. If it is, we need to call ff_mediacodec_dec_flush.
+     *
+     * ff_mediacodec_dec_flush returns 0 if the flush cannot be performed on
+     * the codec (because the user retains frames). The codec stays in the
+     * flushing state.
+     *
+     * ff_mediacodec_dec_flush returns 1 if the flush can actually be
+     * performed on the codec. The codec leaves the flushing state and can
+     * process again packets.
+     *
+     */
+    if (ff_mediacodec_dec_is_flushing(avctx, &s->ctx)) {
+        if (!ff_mediacodec_dec_flush(avctx, &s->ctx)) {
+            return avpkt->size;
+        }
+    }
+
     /* process buffered data */
     while (!*got_frame) {
         /* prepare the input data -- convert to Annex B if needed */
diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
index 981fa0e..8864a3d 100644
--- a/libavutil/pixdesc.c
+++ b/libavutil/pixdesc.c
@@ -1974,6 +1974,10 @@ static const AVPixFmtDescriptor av_pix_fmt_descriptors[AV_PIX_FMT_NB] = {
         .name = "qsv",
         .flags = AV_PIX_FMT_FLAG_HWACCEL,
     },
+    [AV_PIX_FMT_MEDIACODEC] = {
+        .name = "mediacodec",
+        .flags = AV_PIX_FMT_FLAG_HWACCEL,
+    },
     [AV_PIX_FMT_MMAL] = {
         .name = "mmal",
         .flags = AV_PIX_FMT_FLAG_HWACCEL,
diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h
index dbd2470..f8533b0 100644
--- a/libavutil/pixfmt.h
+++ b/libavutil/pixfmt.h
@@ -300,6 +300,8 @@ enum AVPixelFormat {
     AV_PIX_FMT_GBRAP12BE,  ///< planar GBR 4:4:4:4 48bpp, big-endian
     AV_PIX_FMT_GBRAP12LE,  ///< planar GBR 4:4:4:4 48bpp, little-endian
 
+    AV_PIX_FMT_MEDIACODEC, ///< hardware decoding through MediaCodec
+
     AV_PIX_FMT_NB,        ///< number of pixel formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions
 };
 
-- 
2.7.2



More information about the ffmpeg-devel mailing list