[FFmpeg-devel] [PATCHv3 4/4] libavcodec: v4l2: add support for v4l2 mem2mem codecs

[Mark Thompson]

On 02/08/17 08:32, Jorge Ramirez-Ortiz wrote:
> From: Alexis Ballier <aballier at gentoo.org>
> 
> This patchset enhances Alexis Ballier's original patch and validates
> it using Qualcomm's Venus hardware (driver recently landed upstream
> [1]).
> 
> This has been tested on Qualcomm's DragonBoard 410c and 820c
> 
> Tested decoders:
>        - h264
>        - mpeg4
>        - vp8
>        - vp9
>        - hevc
> 
> Tested encoders:
>        -h264
>        -h263
>        -mpeg4
> 
> Tested transcoding (concurrent encoding/decoding)
> 
> Some of the changes introduced:
> - v4l2: code cleanup.
> - v4l2: follow the decode api.
> - v4l2: fix display size for NV12 output pool.
> - v4l2: handle EOS.
> - v4l2: vp8 and mpeg4 decoding.
> - v4l2: hevc and vp9 support.
> - v4l2: generate EOF on dequeue errors.
> - v4l2: h264_mp4toannexb filtering.
> - v4l2: import compat/v4l2 header files.
> - v4l2: fixed make install and fate issues.
> 
> [1] https://lwn.net/Articles/697956/
> 
> Reviewed-by: Jorge Ramirez <jorge.ramirez-ortiz at linaro.org>
> Reviewed-by: Alexis Ballier <aballier at gentoo.org>
> Tested-by: Jorge Ramirez <jorge.ramirez-ortiz at linaro.org>
> ---
>  Changelog                     |    3 +-
>  compat/v4l2/v4l2-common.h     |  107 ++
>  compat/v4l2/v4l2-controls.h   |  987 +++++++++++++++++
>  compat/v4l2/videodev2.h       | 2402 +++++++++++++++++++++++++++++++++++++++++
>  configure                     |   26 +-
>  libavcodec/Makefile           |   18 +-
>  libavcodec/allcodecs.c        |    9 +
>  libavcodec/v4l2-common.c      |  136 ---
>  libavcodec/v4l2-common.h      |   62 --
>  libavcodec/v4l2_buffers.c     |  614 +++++++++++
>  libavcodec/v4l2_buffers.h     |  226 ++++
>  libavcodec/v4l2_fmt.c         |  142 +++
>  libavcodec/v4l2_fmt.h         |   62 ++
>  libavcodec/v4l2_m2m.c         |  356 ++++++
>  libavcodec/v4l2_m2m.h         |   69 ++
>  libavcodec/v4l2_m2m_avcodec.h |   32 +
>  libavcodec/v4l2_m2m_dec.c     |  229 ++++
>  libavcodec/v4l2_m2m_enc.c     |  270 +++++
>  libavdevice/v4l2.c            |    2 +-
>  libavdevice/v4l2enc.c         |    2 +-
>  tests/ref/fate/source         |    3 +
>  21 files changed, 5547 insertions(+), 210 deletions(-)
>  create mode 100644 compat/v4l2/v4l2-common.h
>  create mode 100644 compat/v4l2/v4l2-controls.h
>  create mode 100644 compat/v4l2/videodev2.h
>  delete mode 100644 libavcodec/v4l2-common.c
>  delete mode 100644 libavcodec/v4l2-common.h
>  create mode 100644 libavcodec/v4l2_buffers.c
>  create mode 100644 libavcodec/v4l2_buffers.h
>  create mode 100644 libavcodec/v4l2_fmt.c
>  create mode 100644 libavcodec/v4l2_fmt.h
>  create mode 100644 libavcodec/v4l2_m2m.c
>  create mode 100644 libavcodec/v4l2_m2m.h
>  create mode 100644 libavcodec/v4l2_m2m_avcodec.h
>  create mode 100644 libavcodec/v4l2_m2m_dec.c
>  create mode 100644 libavcodec/v4l2_m2m_enc.c

Hi,

Some general questions first; I'll try to do some detailed review later.

Is the intent here only to really support software memory cases?  That is, to only consider things on the user side and to copy with the CPU between different components.

If not, I don't think the method with only user virtual pointers being visible externally is really the right approach.  For example, mapping to DRM PRIME fds is going to be wanted by a number of use-cases (for import with EGL or other APIs, or display directly via modesetting), and needs the V4L2 buffer metadata for VIDIOC_EXPBUF to work.  To that end, I think it might be better to have an opaque AV_PIX_FMT_V4L2 type carrying the buffer information and managed by the hwcontext API, which can then be mapped to user virtual memory as needed.  Such a setup would support sharing between components inside lav* (e.g. transcode with copy) as well.  Alternatively, it might be possible to use AV_PIX_FMT_DRM_PRIME directly (see the Rockchip codec patchset), though that would exclude drivers not using dma_buf inside the kernel which you suggested previously was a needed case.

Are there more patches to follow?  (Filtering?)  I note that all of the buffer functions in the current version are tagged avpriv, but aren't used outside lavc.

On devices, you mention it's tested on Qualcomm's DragonBoard 410c and 820c; is it expected to work with cores from other vendors?  If I wanted to test this, what hardware would be easiest to use?

Thanks,

- Mark