[Libav-user] HW-accelerated decode on iOS with VideoToolbox?

[Josh Gustafson]

Hi all,

I'm trying to get hardware-accelerated decoding with VideoToolbox working
on iOS. I'm using FFmpeg 3.2, and I'm trying to decode a live TV stream
(streamed from an HD Homerun) with mpegts & h264.

I've got a very simple toy program that works on macOS - see attached
sources. Poc.cpp/h creates and runs the decoder class, and exposes a
function that's callable from C (and Objective-C.) Decoder.cpp/hpp is a
simple C++ class which sets up the appropriate FFmpeg contexts and runs a
simple packet/frame pump. To enable HWA decoding, I set a custom get_format
function on the codec context before opening it; my get_format
implementation then watches to see if AV_PIX_FMT_VIDEOTOOLBOX is an
available option. If so, it calls av_videotoolbox_default_init on the codec
context and returns indicating selection of that format. Again, on macOS
this appears to work as expected, that is, my dummy frame handler function
gets a steady stream of frames, and CPU usage is much lower than it is
without my custom get_format implementation.

However, the same code fails on iOS, and then crashes. Specifically, the
setup code succeeds (or at least, it neither reports nor logs any errors),
then I enter my packet pump, and start pumping a few packets from the demux
to the codec. The first few times, this gets logged:

[h264 @ 0x101982400] Failed to decode frame (-12911)
[h264 @ 0x101982400] hardware accelerator failed to decode picture

...and avcodec_receive_frame returns AVERROR(EAGAIN) (not surprising, as
the codec buffer starts to fill.) Then after a few rounds of that (and
before avcodec_receive_frame actually returns any frames),
avcodec_send_packet crashes, EXC_BAD_ACCESS (code=1, address=0x0) with this
call stack:

#0 0x0000000100274f04 in ff_h264_decode_ref_pic_list_reordering ()
#1 0x000000010027bc58 in ff_h264_decode_slice_header ()
#2 0x000000010023a9c8 in h264_decode_frame ()
#3 0x0000000100596248 in avcodec_decode_video2 ()
#4 0x0000000100597824 in do_decode ()
#5 0x000000010158883c in decoder::run() at
/Users/josh/Git/ffmpeg-sandbox/ffmpeg-poc/ffmpeg-poc/decoder.cpp:85
#6 0x0000000101584e0c in ::run_poc() at
/Users/josh/Git/ffmpeg-sandbox/ffmpeg-poc/ffmpeg-poc/poc.cpp:42

So, is there something that I need to do differently on iOS vs macOS, or is
this a bug, or...?

Or, if nothing else, can anyone point me to an example of working source
code which successfully uses HWA decode on iOS?

*Josh Gustafson*
*Senior Software Engineer*
650.209.0952
www.simplecontrol.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sc_logo_email.gif
Type: image/gif
Size: 3762 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: decoder.cpp
Type: text/x-c++src
Size: 4130 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: decoder.hpp
Type: text/x-c++hdr
Size: 780 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment.hpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: poc.cpp
Type: text/x-c++src
Size: 864 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment-0001.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: poc.h
Type: text/x-chdr
Size: 297 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20161114/93775b1c/attachment.h>