[FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a specific decode format
Mark Thompson
sw at jkqxz.net
Wed Feb 20 23:52:19 EET 2019
On 18/02/2019 05:05, Fu, Linjie wrote:
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
>> Of Fu, Linjie
>> Sent: Friday, November 16, 2018 16:37
>> To: FFmpeg development discussions and patches <ffmpeg-
>> devel at ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>> specific decode format
>>
>>> -----Original Message-----
>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
>> Behalf
>>> Of Mark Thompson
>>> Sent: Thursday, November 15, 2018 05:48
>>> To: ffmpeg-devel at ffmpeg.org
>>> Subject: Re: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>>> specific decode format
>>>
>>> On 14/11/18 01:35, Fu, Linjie wrote:
>>>>> -----Original Message-----
>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
>>> Behalf
>>>>> Of Mark Thompson
>>>>> Sent: Wednesday, November 14, 2018 09:11
>>>>> To: ffmpeg-devel at ffmpeg.org
>>>>> Subject: Re: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>>>>> specific decode format
>>>>>
>>>>> On 14/11/18 00:50, Fu, Linjie wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
>>>>> Behalf
>>>>>>> Of Mark Thompson
>>>>>>> Sent: Wednesday, November 14, 2018 07:44
>>>>>>> To: ffmpeg-devel at ffmpeg.org
>>>>>>> Subject: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>>>>> specific
>>>>>>> decode format
>>>>>>>
>>>>>>> Fixes #7519.
>>>>>>> ---
>>>>>>> doc/ffmpeg.texi | 13 ++++++++++++
>>>>>>> fftools/ffmpeg.c | 10 ++++++++++
>>>>>>> fftools/ffmpeg.h | 4 ++++
>>>>>>> fftools/ffmpeg_opt.c | 47
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> 4 files changed, 74 insertions(+)
>>>>>>>
>>>>>>> diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi
>>>>>>> index 3717f22d42..d127bc0f0d 100644
>>>>>>> --- a/doc/ffmpeg.texi
>>>>>>> +++ b/doc/ffmpeg.texi
>>>>>>> @@ -920,6 +920,19 @@ would be more efficient.
>>>>>>> When doing stream copy, copy also non-key frames found at the
>>>>>>> beginning.
>>>>>>>
>>>>>>> + at item -decode_format[:@var{stream_specifier}]
>>>>>>> @var{pixfmt}[, at var{pixfmt}...] (@emph{input,per-stream})
>>>>>>> +Set the possible output formats to be used by the decoder for this
>>>>> stream.
>>>>>>> +If the decoder does not natively support any format in the given list
>>> for
>>>>>>> +the input stream then decoding will fail rather than continuing with a
>>>>>>> +different format.
>>>>>>> +
>>>>>>> +In general this should not be set - the decoder will select an output
>>>>>>> +format based on the input stream parameters and available
>>> components,
>>>>>>> and
>>>>>>> +that will be automatically converted to whatever the output
>> requires.
>>> It
>>>>>>> +may be useful to force a hardware decoder supporting output in
>>>>> multiple
>>>>>>> +different memory types to pick a desired one, or to ensure that a
>>>>> hardware
>>>>>>> +decoder is used when software fallback is also available.
>>>>>>> +
>>>>>>> @item -init_hw_device
>>>>>>> @var{type}[=@var{name}][:@var{device}[, at var{key=value}...]]
>>>>>>> Initialise a new hardware device of type @var{type} called
>>> @var{name},
>>>>>>> using the
>>>>>>> given device parameters.
>>>>>>> diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
>>>>>>> index 38c21e944a..c651c8d3a8 100644
>>>>>>> --- a/fftools/ffmpeg.c
>>>>>>> +++ b/fftools/ffmpeg.c
>>>>>>> @@ -598,6 +598,7 @@ static void ffmpeg_cleanup(int ret)
>>>>>>> avsubtitle_free(&ist->prev_sub.subtitle);
>>>>>>> av_frame_free(&ist->sub2video.frame);
>>>>>>> av_freep(&ist->filters);
>>>>>>> + av_freep(&ist->decode_formats);
>>>>>>> av_freep(&ist->hwaccel_device);
>>>>>>> av_freep(&ist->dts_buffer);
>>>>>>>
>>>>>>> @@ -2800,6 +2801,15 @@ static enum AVPixelFormat
>>>>>>> get_format(AVCodecContext *s, const enum AVPixelFormat
>>>>>>> const AVCodecHWConfig *config = NULL;
>>>>>>> int i;
>>>>>>>
>>>>>>> + if (ist->decode_formats) {
>>>>>>> + for (i = 0; ist->decode_formats[i] != AV_PIX_FMT_NONE; i++)
>> {
>>>>>>> + if (ist->decode_formats[i] == *p)
>>>>>>> + break;
>>>>>>> + }
>>>>>>> + if (ist->decode_formats[i] != *p)
>>>>>>> + continue;
>>>>>>> + }
>>>>>>> +
>>>>>>> if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL))
>>>>>>> break;
>>>>>>>
>>>>>>> diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h
>>>>>>> index eb1eaf6363..b06fd18b1c 100644
>>>>>>> --- a/fftools/ffmpeg.h
>>>>>>> +++ b/fftools/ffmpeg.h
>>>>>>> @@ -125,6 +125,8 @@ typedef struct OptionsContext {
>>>>>>> int nb_ts_scale;
>>>>>>> SpecifierOpt *dump_attachment;
>>>>>>> int nb_dump_attachment;
>>>>>>> + SpecifierOpt *decode_formats;
>>>>>>> + int nb_decode_formats;
>>>>>>> SpecifierOpt *hwaccels;
>>>>>>> int nb_hwaccels;
>>>>>>> SpecifierOpt *hwaccel_devices;
>>>>>>> @@ -334,6 +336,8 @@ typedef struct InputStream {
>>>>>>> int top_field_first;
>>>>>>> int guess_layout_max;
>>>>>>>
>>>>>>> + enum AVPixelFormat *decode_formats;
>>>>>>> +
>>>>>>> int autorotate;
>>>>>>>
>>>>>>> int fix_sub_duration;
>>>>>>> diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c
>>>>>>> index d4851a2cd8..63bb05053b 100644
>>>>>>> --- a/fftools/ffmpeg_opt.c
>>>>>>> +++ b/fftools/ffmpeg_opt.c
>>>>>>> @@ -701,6 +701,7 @@ static void
>> add_input_streams(OptionsContext
>>> *o,
>>>>>>> AVFormatContext *ic)
>>>>>>> AVStream *st = ic->streams[i];
>>>>>>> AVCodecParameters *par = st->codecpar;
>>>>>>> InputStream *ist = av_mallocz(sizeof(*ist));
>>>>>>> + char *decode_formats = NULL;
>>>>>>> char *framerate = NULL, *hwaccel_device = NULL;
>>>>>>> const char *hwaccel = NULL;
>>>>>>> char *hwaccel_output_format = NULL;
>>>>>>> @@ -797,6 +798,49 @@ static void
>> add_input_streams(OptionsContext
>>>>> *o,
>>>>>>> AVFormatContext *ic)
>>>>>>> ist->top_field_first = -1;
>>>>>>> MATCH_PER_STREAM_OPT(top_field_first, i, ist-
>>> top_field_first,
>>> ic,
>>>>>>> st);
>>>>>>>
>>>>>>> + MATCH_PER_STREAM_OPT(decode_formats, str,
>>>>> decode_formats, ic,
>>>>>>> st);
>>>>>>> + if (decode_formats) {
>>>>>>> + const char *p, *q;
>>>>>>> + int i, nb_formats;
>>>>>>> + char tmp[32];
>>>>>>> +
>>>>>>> + nb_formats = 0;
>>>>>>> + for (p = decode_formats; p; p = strchr(p + 1, ','))
>>>>>>> + ++nb_formats;
>>>>>>> +
>>>>>>> + ist->decode_formats =
>>>>>>> + av_malloc_array(nb_formats + 1, sizeof(*ist-
>>>>>> decode_formats));
>>>>>>> + if (!ist->decode_formats)
>>>>>>> + exit_program(1);
>>>>>>> +
>>>>>>> + p = decode_formats;
>>>>>>> + for (i = 0; i < nb_formats; i++) {
>>>>>>> + q = strchr(p, ',');
>>>>>>> + if (!q) {
>>>>>>> + ist->decode_formats[i] = av_get_pix_fmt(p);
>>>>>>> + if (ist->decode_formats[i] == AV_PIX_FMT_NONE)
>>>>>>> + break;
>>>>>>> + continue;
>>>>>>> + }
>>>>>>> + if (q - p > sizeof(tmp) - 1)
>>>>>>> + break;
>>>>>>> + memcpy(tmp, p, q - p);
>>>>>>> + tmp[q - p] = 0;
>>>>>>> + ist->decode_formats[i] = av_get_pix_fmt(tmp);
>>>>>>> + if (ist->decode_formats[i] == AV_PIX_FMT_NONE)
>>>>>>> + break;
>>>>>>> + p = q + 1;
>>>>>>> + }
>>>>>>> + if (i < nb_formats) {
>>>>>>> + av_log(NULL, AV_LOG_FATAL,
>>>>>>> + "Unrecognised decode format: %s.\n", p);
>>>>>>> + exit_program(1);
>>>>>>> + }
>>>>>>> + ist->decode_formats[nb_formats] = AV_PIX_FMT_NONE;
>>>>>>> + } else {
>>>>>>> + ist->decode_formats = NULL;
>>>>>>> + }
>>>>>>> +
>>>>>>> MATCH_PER_STREAM_OPT(hwaccels, str, hwaccel, ic, st);
>>>>>>> if (hwaccel) {
>>>>>>> // The NVDEC hwaccels use a CUDA device, so remap the
>>> name
>>>>> here.
>>>>>>> @@ -3583,6 +3627,9 @@ const OptionDef options[] = {
>>>>>>> "audio bitrate (please use -b:a)", "bitrate" },
>>>>>>> { "b", OPT_VIDEO | HAS_ARG | OPT_PERFILE | OPT_OUTPUT,
>>>>>>> { .func_arg = opt_bitrate },
>>>>>>> "video bitrate (please use -b:v)", "bitrate" },
>>>>>>> + { "decode_format", OPT_VIDEO | OPT_STRING | HAS_ARG |
>>>>>>> OPT_EXPERT |
>>>>>>> + OPT_SPEC | OPT_INPUT, { .off =
>>>>>>> OFFSET(decode_formats) },
>>>>>>> + "set output format(s) to be used by decoder, fail if none of
>> these
>>>>>>> formats are available", "format" },
>>>>>>> { "hwaccel", OPT_VIDEO | OPT_STRING | HAS_ARG |
>>> OPT_EXPERT
>>>>> |
>>>>>>> OPT_SPEC | OPT_INPUT, { .off =
>>>>>>> OFFSET(hwaccels) },
>>>>>>> "use HW accelerated decoding", "hwaccel name" },
>>>>>>> --
>>>>>>> 2.19.1
>>>>>>>
>>>>>>
>>>>>> Is there any other conditions which will cause the hwaccel failed and
>>>>> fallback to software path?
>>>>>> Like profile was not supported by hardware?
>>>>>
>>>>> If this option is set then get_format will never choose a format not on
>> the
>>> list
>>>>> you provide, so it shouldn't be able to use software decode unless a
>>>>> software format is on the list.
>>>>>
>>>>>> I applied the patch and tested some cases, like yuv422 VAAPI
>> decoding
>>>>> (currently not supported), HEVC decoding with unsupported profile.
>>>>>> It reports an error like "hwaccel initialization returned error", and
>>> continue
>>>>> to decode and produce less frames than expected. (which I thought
>>> fallback
>>>>> still happened).
>>>>>
>>>>> Can you give an example? With -v debug there will be output saying
>>> exactly
>>>>> what happened in ff_get_format().
>>>>>
>>>>> - Mark
>>>>
>>>>
>>>> Got information like this:
>>>>
>>>> [hevc @ 0x55b87457dbc0] Unknown HEVC profile: 0
>>>> [hevc @ 0x55b87457dbc0] Decoding SPS
>>>> [hevc @ 0x55b87457dbc0] Unknown HEVC profile: 0
>>>> [hevc @ 0x55b87457dbc0] Decoding PPS
>>>> [hevc @ 0x55b87457dbc0] Format vaapi_vld chosen by get_format().
>>>> [hevc @ 0x55b87457dbc0] Format vaapi_vld requires hwaccel initialisation.
>>>> [hevc @ 0x55b87457dbc0] Codec hevc profile 0 not supported for
>> hardware
>>> decode.
>>>> [hevc @ 0x55b87457dbc0] Failed setup for format vaapi_vld: hwaccel
>>> initialisation returned error.
>>>> [hevc @ 0x55b87457dbc0] Format vaapi_vld not usable, retrying
>>> get_format() without it.
>>>> [hevc @ 0x55b87457dbc0] Error parsing NAL unit #3.
>>>> cur_dts is invalid (this is harmless if it occurs once at the start per stream)
>>>> [hevc @ 0x55b87458b7c0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>> temporal_id: 0
>>>> [hevc @ 0x55b87458b7c0] Could not find ref with POC 0
>>>>
>>>> See the attached report file for more details.
>>>
>>> This is a decoder bug - get_format() is not correctly called again after it fails,
>>> and the decoder then uses the software pixfmt derived from the SPS on
>>> subsequent frames.
>>>
>>> (I did pretty much all of my testing with H.264, which does not have this
>>> problem.)
>>>
>> Works for me on forcing hwaccel in HEVC vaapi decoding after applying the
>> hevcdec patch.
>> Thanks for answering my questions.
>>
> Ping?
> Can this be merged?
> https://trac.ffmpeg.org/ticket/7519#comment:4
I think this got lost because I was feeling it loosely depended on <https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2018-November/236376.html> in order to actually work in the intended use-case. I wasn't particularly confident in that patch being the best way to do it and it isn't my code, so I was hoping for any thoughts from someone more familiar with it.
Shall I apply this part anyway?
Thanks,
- Mark
More information about the ffmpeg-devel
mailing list