[FFmpeg-devel] [PATCHv3 4/4] libavcodec: v4l2: add support for v4l2 mem2mem codecs

Tue Aug 8 12:30:38 EEST 2017

On 08/08/17 09:33, Jorge Ramirez wrote:
> On 08/08/2017 10:23 AM, Martin Storsjö wrote:
>>>>>
>>>> This doesn't set extradata - you need to extract the codec global headers 
>>> (such as H.264 SPS and PPS) at init time to be able to write correct files for some codecs (such as H.264) with muxers requiring global headers (such as MP4).  It kindof works without it, but the files created will not conform and will not be usable on some players.
>>>>
>>> ah that might explain some things (when I play back the encoded video the quality is pretty lousy)
>>> is there already some code I can use as a reference? I might be out of my depth here so any help will be more than welcome
>>
>> This is exactly the thing I was trying to tell you about, off list, before.
> 
> Hi Martin,
> 
> yes sorry I was just thinking about that. I was going to follow up but little details here on this patchset keep on piling up.
> 
> btw -just for info- the encoding issues I was seeing were related to the time stamps not being properly passed to the encoder...still trying to figure out that as well but at least I now get a proper encoded/decoded frame (even though just one, that keeps on being played when I reproduce the encoded video)
> 
> the encoder however does not complain during the process so I assume the other frames are also there
> 
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.63e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.61e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.6e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.58e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.56e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.55e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.53e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.52e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.5e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.48e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.47e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.45e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.44e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.42e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.41e-07x
> [rawvideo @ 0xaaaae5e600a0] PACKET SIZE: 1382400, STRIDE: 1920trate=4923.1kbits/s speed=3.39e-07x
> [out_0_0 @ 0xaaaae5e64be0] EOF on sink link out_0_0:default.bitrate=4923.1kbits/s speed=3.36e-07x
> 
> No more output streams to write to, finishing.
> frame=  232 fps=1.0 q=-0.0 Lsize=      42kB time=00:00:00.00 bitrate=736989.3kbits/s speed=2.01e-06x
> video:41kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.783316%
> Input file #0 (/home/linaro/Videos/raw/freeway.yuv):
>   Input stream #0:0 (video): 232 packets read (320716800 bytes); 232 frames decoded;
>   Total: 232 packets (320716800 bytes) demuxed
> Output file #0 (out/out.h264.mp4):
>   Output stream #0:0 (video): 232 frames encoded; 6 packets muxed (42449 bytes);
>   Total: 6 packets (42449 bytes) muxed
> 232 frames successfully decoded, 0 decoding errors
> 
>>
>> In the OMX driver used on android, this is requested on startup, via an ioctl with the following private ioctl value:
>> V4L2_CID_MPEG_VIDC_VIDEO_REQUEST_SEQ_HEADER
>>
>> See this code here:
>> https://android.googlesource.com/platform/hardware/qcom/media/+/63abe022/msm8996/mm-video-v4l2/vidc/venc/src/video_encoder_device_v4l2.cpp#2991
>>
>> This is a qcom specific, private ioctl. In the android kernel for qualcomm, this is handled correctly here:
>>
>> https://android.googlesource.com/kernel/msm/+/android-7.1.2_r0.33/drivers/media/platform/msm/vidc/msm_venc.c#2987
>> https://android.googlesource.com/kernel/msm/+/android-7.1.2_r0.33/drivers/media/platform/msm/vidc/msm_vidc_common.c#3767
>>
>> In the dragonboard kernel snapshot I had been testing, that I referred to you before, there are incomplete stubs of handling of this. In the debian-qcom-dragonboard410c-16.04 tag in the linaro kernel tree:
>>
>> http://git.linaro.org/landing-teams/working/qualcomm/kernel.git/tree/drivers/media/platform/msm/vidc/msm_venc-ctrls.c?h=debian-qcom-dragonboard410c-16.04&id=8205f603ceeb02d08a720676d9075c9e75e47b0f#n2116
>> This increments seq_hdr_reqs, just like in the android kernel tree (where this is working). However in this kernel tree, nothing actually ever reads the seq_hdr_reqs, so it's a non-functional stub.
> 
> ok. thanks for the info
> 
>>
>> Now in the kernel tree you referred me to, in the release/db820c/qcomlt-4.11 branch, I don't see anything similar to V4L2_CID_MPEG_VIDC_VIDEO_REQUEST_SEQ_HEADER. I can't help you from there, you need to figure that out what alternative codepath there is, intended to replace it - if any. If there aren't any, you first need to fix the v4l2 driver before userspace apps can get what they need.
> 
> many thanks for all the details (extremely helpful)
> Just added  Stanimir to this thread so he can look at it once he is back from his break.
> 
>>
>> There is a clear need for this, as you witness in the android version of the kernel. It just seems to have been removed in the vanilla linux version of the driver.
> 
> understood.

Note that while this may be blocking for all ITU/ISO codecs, you can still support VP8/VP9 without it.

- Mark