[FFmpeg-devel] [PATCHv3 4/4] libavcodec: v4l2: add support for v4l2 mem2mem codecs

Jorge Ramirez jorge.ramirez-ortiz at linaro.org
Wed Aug 2 15:53:45 EEST 2017


On 08/02/2017 12:16 PM, Mark Thompson wrote:
> On 02/08/17 08:32, Jorge Ramirez-Ortiz wrote:
>> From: Alexis Ballier <aballier at gentoo.org>
>>
>> This patchset enhances Alexis Ballier's original patch and validates
>> it using Qualcomm's Venus hardware (driver recently landed upstream
>> [1]).
>>
>> This has been tested on Qualcomm's DragonBoard 410c and 820c
>>
>> Tested decoders:
>>         - h264
>>         - mpeg4
>>         - vp8
>>         - vp9
>>         - hevc
>>
>> Tested encoders:
>>         -h264
>>         -h263
>>         -mpeg4
>>
>> Tested transcoding (concurrent encoding/decoding)
>>
>> Some of the changes introduced:
>> - v4l2: code cleanup.
>> - v4l2: follow the decode api.
>> - v4l2: fix display size for NV12 output pool.
>> - v4l2: handle EOS.
>> - v4l2: vp8 and mpeg4 decoding.
>> - v4l2: hevc and vp9 support.
>> - v4l2: generate EOF on dequeue errors.
>> - v4l2: h264_mp4toannexb filtering.
>> - v4l2: import compat/v4l2 header files.
>> - v4l2: fixed make install and fate issues.
>>
>> [1] https://lwn.net/Articles/697956/
>>
>> Reviewed-by: Jorge Ramirez <jorge.ramirez-ortiz at linaro.org>
>> Reviewed-by: Alexis Ballier <aballier at gentoo.org>
>> Tested-by: Jorge Ramirez <jorge.ramirez-ortiz at linaro.org>
>> ---
>>   Changelog                     |    3 +-
>>   compat/v4l2/v4l2-common.h     |  107 ++
>>   compat/v4l2/v4l2-controls.h   |  987 +++++++++++++++++
>>   compat/v4l2/videodev2.h       | 2402 +++++++++++++++++++++++++++++++++++++++++
>>   configure                     |   26 +-
>>   libavcodec/Makefile           |   18 +-
>>   libavcodec/allcodecs.c        |    9 +
>>   libavcodec/v4l2-common.c      |  136 ---
>>   libavcodec/v4l2-common.h      |   62 --
>>   libavcodec/v4l2_buffers.c     |  614 +++++++++++
>>   libavcodec/v4l2_buffers.h     |  226 ++++
>>   libavcodec/v4l2_fmt.c         |  142 +++
>>   libavcodec/v4l2_fmt.h         |   62 ++
>>   libavcodec/v4l2_m2m.c         |  356 ++++++
>>   libavcodec/v4l2_m2m.h         |   69 ++
>>   libavcodec/v4l2_m2m_avcodec.h |   32 +
>>   libavcodec/v4l2_m2m_dec.c     |  229 ++++
>>   libavcodec/v4l2_m2m_enc.c     |  270 +++++
>>   libavdevice/v4l2.c            |    2 +-
>>   libavdevice/v4l2enc.c         |    2 +-
>>   tests/ref/fate/source         |    3 +
>>   21 files changed, 5547 insertions(+), 210 deletions(-)
>>   create mode 100644 compat/v4l2/v4l2-common.h
>>   create mode 100644 compat/v4l2/v4l2-controls.h
>>   create mode 100644 compat/v4l2/videodev2.h
>>   delete mode 100644 libavcodec/v4l2-common.c
>>   delete mode 100644 libavcodec/v4l2-common.h
>>   create mode 100644 libavcodec/v4l2_buffers.c
>>   create mode 100644 libavcodec/v4l2_buffers.h
>>   create mode 100644 libavcodec/v4l2_fmt.c
>>   create mode 100644 libavcodec/v4l2_fmt.h
>>   create mode 100644 libavcodec/v4l2_m2m.c
>>   create mode 100644 libavcodec/v4l2_m2m.h
>>   create mode 100644 libavcodec/v4l2_m2m_avcodec.h
>>   create mode 100644 libavcodec/v4l2_m2m_dec.c
>>   create mode 100644 libavcodec/v4l2_m2m_enc.c
> Hi,
>
> Some general questions first; I'll try to do some detailed review later.

thanks! very much appreciated.

Alexis feel free to add your comments as well.

>
> Is the intent here only to really support software memory cases?  That is, to only consider things on the user side and to copy with the CPU between different components.

This commit adds a generic way of supporting v4l2 in ffmpeg 
(V4L2_MEMORY_MMAP)

enum v4l2_memory {
     V4L2_MEMORY_MMAP             = 1,
     V4L2_MEMORY_USERPTR          = 2,
     V4L2_MEMORY_OVERLAY          = 3,
     V4L2_MEMORY_DMABUF           = 4,
};

You are right, this patch uses the CPU to copy the preprocessed/demuxed 
bitstream to the v4l2 buffers and then as you said uses the pointers.

Extending it to support V4L2_MEMORY_DMABUF for performance is in the 
todo list.
however I think (this is just my opinion) there is value today in 
enabling the available v4l2 codecs present in many SoCs and bringing 
them under ffmpeg.
And right after work on performance and incremental optimizations.

>
> If not, I don't think the method with only user virtual pointers being visible externally is really the right approach.

if by "right" you mean with the best performance possible yes I don't 
disagree with you.

> For example, mapping to DRM PRIME fds is going to be wanted by a number of use-cases (for import with EGL or other APIs, or display directly via modesetting), and needs the V4L2 buffer metadata for VIDIOC_EXPBUF to work.  To that end, I think it might be better to have an opaque AV_PIX_FMT_V4L2 type carrying the buffer information and managed by the hwcontext API, which can then be mapped to user virtual memory as needed.

yes I am looking at this WIP 
https://github.com/LongChair/FFmpeg/commit/f852e3d6d89be3fcee90482971c3f769bdaa8e9c. 

I think it is work like this that can be used later on with v4l2 for 
optimal performance...thought I have to admit I still need to understand 
how I would use it.
but I don't see why the v4l2 patchset can not be extended afterwards?

Yes we need to add VIDIOC_EXPBUF and then on dequeing cacpture we coud 
either set the DRM plane or just pass the pointer.


> Such a setup would support sharing between components inside lav* (e.g. transcode with copy) as well.  Alternatively, it might be possible to use AV_PIX_FMT_DRM_PRIME directly (see the Rockchip codec patchset), though that would exclude drivers not using dma_buf inside the kernel which you suggested previously was a needed case.
>
> Are there more patches to follow?  (Filtering?)  I note that all of the buffer functions in the current version are tagged avpriv, but aren't used outside lavc.

yes filtering v4l2_m2m filtering is queued after this.
I just didn't want to add too much noise....

>
> On devices, you mention it's tested on Qualcomm's DragonBoard 410c and 820c; is it expected to work with cores from other vendors?  If I wanted to test this, what hardware would be easiest to use?

yes this should to work on _any_ SoC that provides a v4l2 driver 
implementing either encoding or decoding. Lukas Rusak integrated this 
ffmpeg release (with the patchset) with Kodi and was able to run on i.MX6
I just happen to have access to the Qualcomm boards really

>
> Thanks,
>
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel



More information about the ffmpeg-devel mailing list