[FFmpeg-user] Intel QSV and xstack_qsv: Error while filtering: Internal bug, should not have happened

Xiang, Haihao haihao.xiang at intel.com
Fri Dec 29 05:07:30 EET 2023 

On Do, 2023-12-28 at 14:46 +0000, Shane Warren wrote:
> It must be in the latest ffmpeg because I took all the hwupload and hwdownload
> out of my command and everything still works.
> 
> One question I have is:
> 
> Why is hwdownload not needed before the pad filter and hwupload needed after
> the pad filter. Before trying out intel's hw transcoding I used nvidia's cards
> and as far as I remember one always had to hwdownload when doing cpu filters
> and hwupload when going back to gpu filters or gpu encoding.

vpp_qsv filter supports output in system memory and h264_qsv encoder supports
input in system memory too, so you may use or not use hwdownload and hwupload in
your case. 

[...]
[Parsed_vpp_qsv_0 @ 0x7f07b00065d0] VPP: input is video memory surface
[Parsed_vpp_qsv_0 @ 0x7f07b00065d0] VPP: output is video memory surface
[...]
[h264_qsv @ 0x5636855b0f80] Encoder: input is system memory surface

Thanks
Haihao


> 
> Anyway, thanks for the help, updated command survives resolution changes.
> 
> ________________________________
> From: ffmpeg-user <ffmpeg-user-bounces at ffmpeg.org> on behalf of Dennis Mungai
> <dmngaie at gmail.com>
> Sent: Thursday, December 28, 2023 1:23 AM
> To: FFmpeg user questions <ffmpeg-user at ffmpeg.org>
> Cc: artem.galin at gmail.com <artem.galin at gmail.com>; Wang, Fei W
> <fei.w.wang at intel.com>
> Subject: Re: [FFmpeg-user] Intel QSV and xstack_qsv: Error while filtering:
> Internal bug, should not have happened
> 
> 
> On Thu, 28 Dec 2023, 05:15 Xiang, Haihao, <
> haihao.xiang-at-intel.com at ffmpeg.org> wrote:
> 
> > 
> > > I have created a bug and linked a sample video that changes resolutions
> > on the
> > > fly:
> > > 
> > > https://trac.ffmpeg.org/ticket/10762<https://trac.ffmpeg.org/ticket/10762>
> > > 
> > > Get back to me if you need anything else I can help with.
> > > ________________________________
> > > From: ffmpeg-user <ffmpeg-user-bounces at ffmpeg.org> on behalf of Dennis
> > Mungai
> > > <dmngaie at gmail.com>
> > > Sent: Wednesday, December 27, 2023 11:38 AM
> > > To: FFmpeg user questions <ffmpeg-user at ffmpeg.org>
> > > Cc: Haihao <haihao.xiang at intel.com>;
> > > artem.galin at gmail.com <artem.galin at gmail.com>;
> > > fei.w.wang at intel.com <fei.w.wang at intel.com>
> > > Subject: Re: [FFmpeg-user] Intel QSV and xstack_qsv: Error while
> > filtering:
> > > Internal bug, should not have happened
> > > 
> > > [You don't often get email from dmngaie at gmail.com. Learn why this is
> > important
> > > at https://aka.ms/LearnAboutSenderIdentification ]
> > > 
> > > This email originated from outside Innovative Systems. Do not click
> > links or
> > > open attachments unless you recognize the sender and know the content is
> > safe.
> > > 
> > > 
> > > On Wed, 27 Dec 2023 at 18:40, Shane Warren <shanew at innovsys.com> wrote:
> > > 
> > > > 
> > > > I'm testing out an Intel Flex 140 card and I am attempting to
> > transcode 4
> > > > inputs to 1 output using the xstack_qsv filter.  My command takes in 4
> > udp
> > > > multicast ts files (h264 encoded) and outputs one h264_qsv encoded udp
> > > > transport stream. My command looks like so:
> > > > 
> > > > /tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device
> > > > qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1
> > 
> > Please use a render node instead, and you should use the key of
> > child_device if
> > you want to use a non-default render node:
> > 
> > qsv=card1,child_device=/dev/dri/renderD129 -hwaccel_device card1 -
> > filter_hw_device card1
> > 
> > 
> > > > -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  \
> > > > -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> > > > -async_depth 1 -i "udp://@
> > > > 
> > 225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun
> > > > _nonfatal=1"
> > > > \
> > > > -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> > > > -async_depth 1 -i "udp://@
> > > > 
> > 225.105.0.14:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overru
> > > > n_nonfatal=1"
> > > > \
> > > > -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> > > > -async_depth 1 -i "udp://@
> > > > 
> > 225.105.0.5:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun
> > > > _nonfatal=1"
> > > > \
> > > > -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> > > > -async_depth 1 -i "udp://@
> > > > 
> > 225.105.0.4:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun
> > > > _nonfatal=1"
> > > > \
> > > > -noautoscale -filter_complex "\
> > > > [0:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v1]; \
> > > > [1:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v2]; \
> > > > [2:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v3]; \
> > > > [3:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v4]; \
> > > > [v1][v2][v3][v4]
> > xstack_qsv=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[mosiac]; \
> > > > [mosiac]vpp_qsv=w=1920:h=1080,hwdownload,format=nv12,pad=1920:1080:(ow-
> > > > iw)/2:(oh-ih)/2,hwupload=extra_hw_frames=64,format=qsv[v]"
> > 
> > -c:v h264_qsv encoder can accept data in system memory, you may remove
> > 'hwupload=extra_hw_frames=64,format=qsv' from the filtergraph.
> > 
> > If you want to use hwupload filter to upload data from system memory to
> > video
> > memory before -c:v h264_qsv encoder, could you try with
> > https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=10318<
> > https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=10318> which
> > added the
> > support for dynamic frame pool in qsv (Note a newer version of
> > oneVPL-intel-gpu
> > is required, you may download oneVPL-intel-gpu from
> > https://github.com/oneapi-src/oneVPL-intel-gpu/tags<
> > https://github.com/oneapi-src/oneVPL-intel-gpu/tags> , and should not use
> > extra_hw_frame=64 anymore )
> > 
> > Thanks
> > Haihao
> > 
> > 
> > > > \
> > > > -map "[v]" -map 0:a:0 -map 1:a:0 -map 2:a:0 -map 3:a:0 \
> > > > -c:v h264_qsv -noautoscale -async_depth 1 -scenario livestreaming -r:v
> > > > 30000/1001 -b:v 6000k -minrate:v 6000k -maxrate:v 6000k -bufsize:v
> > 12000k
> > > > -threads 1 -profile:v high -bf:v 0 -g:v 15 \
> > > > -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k
> > > > -fps_mode auto \
> > > > -f mpegts -muxrate 7450599 -pes_payload_size 1528 "udp://@
> > > > 
> > 229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=7450599&burst_bit
> > > > s=10528
> > > > "
> > > > 
> > > > The issue is eventually stream 1 changes resolution from 1280x720
> > > > (progressive) to 1920x1080 (interlaced). The stream dies with the
> > following
> > > > logs coming out:
> > > > 
> > > > [h264_qsv @ 0x55a240e68000] Error submitting the frame for encoding.
> > > > [vost#0:0/h264_qsv @ 0x55a240e30780] Error submitting video frame to
> > the
> > > > encoder
> > > > Error while filtering: Internal bug, should not have happened
> > > > 
> > > > I can transcode the single stream that changes resolution on its own
> > and
> > > > it can survive a resolution change using this command:
> > > > 
> > > > /tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device
> > > > qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1
> > > > -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt
> > -hwaccel
> > > > qsv -hwaccel_output_format qsv -async_depth 1 -i "udp://@
> > > > 
> > 225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun
> > > > _nonfatal=1"
> > > > -vf "vpp_qsv=deinterlace=2:w=1920:h=1080:framerate=30000/1001" -map
> > "0:v"
> > > > -map 0:a:0 -c:v h264_qsv -noautoscale  -async_depth 1 -scenario
> > > > livestreaming -r:v 30000/1001 -b:v 5000k -minrate:v 5000k -maxrate:v
> > 5000k
> > > > -bufsize:v 10000k -a53cc 1 -forced_idr 1 -threads 1 -profile:v high
> > -bf:v 0
> > > > -g:v 15 -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a
> > > > 192k -fps_mode auto -f mpegts -muxrate 6450599 -pes_payload_size 1528
> > > > "udp://@
> > > > 
> > 229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=6450599&burst_bit
> > > > s=10528
> > > > "
> > > > 
> > > > When that stream changes resolution I see these come out in the log,
> > and
> > > > the stream carries on without dying:
> > > > 
> > > > [h264_qsv @ 0x561bc472f8c0] Video parameter change
> > > > [AVHWDeviceContext @ 0x7f313000d1c0] VAAPI driver: Intel iHD driver for
> > > > Intel(R) Gen Graphics - 23.2.4 ().
> > > > [AVHWDeviceContext @ 0x7f313000d1c0] Driver not found in known
> > nonstandard
> > > > list, using standard behaviour.
> > > > [h264_qsv @ 0x561bc472f8c0] Decoder: output is video memory surface
> > > > 
> > > > My quesiton is why does the stream die when using xstack_qsv but works
> > > > fine on its own.
> > > > 
> > > 
> > > This is definitely a bug, something to do with frame pool setting(s) in
> > > that filter chain. You may want to open a ticket for this on ffmpeg trac.
> > > The xstack_qsv may need a dynamic frame pool workaround to address this
> > > anomaly on video parameter changes.
> > > Adding in the Intel guys on this one.
> > 
> 
> 
> Is this patchwork
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=10318<
> https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=10318> already
> present in ffmpeg-cartwheel?
> 
> > 
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user<
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user>
> 
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> 
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".

More information about the ffmpeg-user mailing list