[FFmpeg-devel] [PATCH 5/8] lavd: add device capabilities API

[Lukasz Marek]

On 04.04.2014 22:01, Michael Niedermayer wrote:
> On Fri, Apr 04, 2014 at 08:23:48PM +0200, Lukasz Marek wrote:
>> On 4 April 2014 19:28, Michael Niedermayer <michaelni at gmx.at> wrote:
>>>
>>>>   /**
>>>> @@ -228,6 +231,128 @@ int avdevice_dev_to_app_control_message(struct
>>> AVFormatContext *s,
>>>>                                           void *data, size_t data_size);
>>>>
>>>>   /**
>>>> + * Following API allows user to probe device capabilities (supported
>>> codecs,
>>>> + * pixel formats, sample formats, resolutions, channel counts, etc).
>>>> + * It is build on top op AVOption API.
>>>> + * Queried capabilities allows to set up converters of video or audio
>>>> + * parameters that fit to the device.
>>>> + *
>>>> + * List of capablities that can be queried:
>>>> + *  - Capabilities valid for both audio and video devices:
>>>> + *    - codec:          supported audio/video codecs.
>>>> + *                      type: AV_OPT_TYPE_INT (AVCodecID value)
>>>> + *    - format:         supported pixel/sample formats.
>>>> + *                      type: AV_OPT_TYPE_INT (AVPixelFormat or
>>> AVSampleFormat value)
>>>> + *  - Capabilities valid for audio devices:
>>>> + *    - sample_rate:    supported sample rates.
>>>> + *                      type: AV_OPT_TYPE_INT
>>>> + *    - channels:       supported number of channels.
>>>> + *                      type: AV_OPT_TYPE_INT
>>>> + *    - channel_layout: supported channel layouts.
>>>> + *                      type: AV_OPT_TYPE_INT64
>>>> + *  - Capabilities valid for audio devices:
>>>> + *    - window_size:    supported window sizes (describes size of the
>>> window size presented to the user).
>>>> + *                      type: AV_OPT_TYPE_IMAGE_SIZE
>>>> + *    - frame_size:     supported frame sizes (describes size of
>>> provided video frames).
>>>> + *                      type: AV_OPT_TYPE_IMAGE_SIZE
>>>> + *    - fps:            supported fps values
>>>> + *                      type: AV_OPT_TYPE_RATIONAL
>>>> + *
>>>> + * Value of the capability may be set by user using av_opt_set()
>>> function
>>>> + * and AVDeviceCapabilitiesQuery object. Following queries will
>>>> + * limit results to the values matching already set capabilities.
>>>> + * For example, setting a codec may impact number of formats or fps
>>> values
>>>> + * returned during next query. Setting invalid value may limit results
>>> to zero.
>>>> + *
>>>> + * Example of the usage basing on opengl output device:
>>>> + *
>>>
>>>> + * @code
>>>> + *  AVFormatContext *oc = NULL;
>>>> + *  AVDeviceCapabilitiesQuery *caps = NULL;
>>>> + *  AVOptionRanges *ranges;
>>>> + *  int ret;
>>>> + *
>>>> + *  if ((ret = avformat_alloc_output_context2(&oc, NULL, "opengl",
>>> NULL)) < 0)
>>>> + *      goto fail;
>>>
>>>
>>>> + *  if (avdevice_capabilities_create(&caps, oc, NULL) < 0)
>>>> + *      goto fail;
>>>
>>> can this be called multiple times ?
>>> if not, why is a free function needed ?
>>> caps could just be kept track of and freed in avformat_free_context()
>>>
>>
>> I simple case there is no need to call it more than once.
>> I don't know if there is a useful use case that would need more.
>>
>>> + *
>>>> + *  //query codecs
>>>> + *  if (av_opt_query_ranges(&ranges, caps, "codec",
>>> AV_OPT_MULTI_COMPONENT_RANGE)) < 0)
>>>> + *      goto fail;
>>>> + *  //pick codec here and set it
>>>> + *  av_opt_set(caps, "codec", AV_CODEC_ID_RAWVIDEO, 0);
>>>> + *
>>>> + *  //query format
>>>> + *  if (av_opt_query_ranges(&ranges, caps, "format",
>>> AV_OPT_MULTI_COMPONENT_RANGE)) < 0)
>>>> + *      goto fail;
>>>> + *  //pick format here and set it
>>>> + *  av_opt_set(caps, "format", AV_PIX_FMT_YUV420P, 0);
>>>> + *
>>>> + *  //query and set more capabilities
>>>> + *
>>>> + * fail:
>>>> + *  //clean up code
>>>> + *  avdevice_capabilities_free(&query, oc);
>>>> + *  avformat_free_context(oc);
>>>
>>> naively i would have thought it could be done like this:
>>>
>>> if ((ret = avformat_alloc_output_context2(&oc, NULL, "opengl", NULL)) < 0)
>>>      goto fail;
>>>
>>> if (av_opt_query_ranges(&ranges, oc, "codec",
>>> AV_OPT_MULTI_COMPONENT_RANGE)) < 0)
>>>      goto fail;
>>> av_opt_set(caps, "codec", AV_CODEC_ID_RAWVIDEO, 0);
>>>
>>> if (av_opt_query_ranges(&ranges, oc, "format",
>>> AV_OPT_MULTI_COMPONENT_RANGE)) < 0)
>>>      goto fail;
>>> av_opt_set(caps, "format", AV_PIX_FMT_YUV420P, 0);
>>>
>>> but maybe this has some issue, can you explain why a seperate
>>> "external" struct wth alloc and apply functions is needed ?
>>>
>>
>> This is similar to one of previous versions I posted.
>> http://ffmpeg.org/pipermail/ffmpeg-devel/2014-February/153977.html
>>
>> I don't remember now all problems it caused, but you need add AVOptions
>> that are probed with this API to device options.
>> They will be visible to the user and may confuse. Also some devices have
>> some of this options already, sometimes under different name etc.
>> So this is one problem I remember.
>
> 2 devices which export the same thing under different names should be
> fixed by adding a "alias" to one of them so a common identifer can
> be used to access the option for both
> (thats quite orthogonal to the subject though)
>
> about the caps stuff
> please correct me if iam wrong
> but with the patch there would be 2 systems
> one would allow setting width, fps, height, ... through
> AVFormatContext and would allow querrying valid ranges of these
> through AVOption / AVOptionRanges.
>
> and the other would allow allocating the seperate AVDeviceCapabilitiesQuery
> struct in which  options
> can be set and their valid ranges querried
> and these can then be applied to the AVFormatContext
>
> the first case is currently implemented in applications, and can be
> accessed from the command line
>
> I suspect also the 2 variants would give different AVOptionRanges
> when querried for the same field like width, where only the
> AVDeviceCapabilitiesQuery would give the range actually supported
>
> This seems somewhat confusing

I think we have to distinguish input and output devices.
I designed it having mostly output devices in mind and it is not optimal 
for input devices. Things you point are 100% true for input devices and 
hardly true for output devices.

Probably any output device has an option that user want to query. Output 
devices doesn't have options like codec, format, frame_size etc because 
they take them from provided stream. User want to query them to make 
sure they provide correct stream. I wanted to avoid adding these options 
to device's option list because they are only useful during probing, not 
during normal use of the device. When such options appears on the list, 
they may confuse the user (for example when they list options using 
av_opt_next() function)

For input devices everything is opposite. Probably all devices have the 
same options already on its options list, so your remarks are OK.

I think it would be good to implement it different for input and output 
devices (I don't mean separate API, but rather different approach when 
implementing the device). input devices would use its own options, 
output devices would use the structure I added and hide its options from 
user.

I just don't know how to implement it yet.
Maybe additional flag for av_opt_query_ranges to mark that capabilities 
are queried, other function I already mentioned in previous email, or 
resign from hiding dummy options. I dont like these dummy options 
visible tho.

At this moment it would be the best to have API call like this:

av_opt_query_ranges(&ranges, oc, "format", AV_OPT_MULTI_COMPONENT_RANGE 
| AV_OPT_DEVICE_CAPABILITY)

callback in AVFormatContext may pass it further to its children when 
querying input device, and or pass it to AVDeviceCapabilitiesQuery when 
querying output device.

There would be no other API required to use when 
AVDeviceCapabilitiesQuery is allocated and freed by already existing 
methods to allocate AVFormatContex.


>> Also, when new caps are added they would need updates everywhere, now just
>> query method need update (not very significant, but still easier to
>> maintain)
>>
>> One solution I see now is to add AVDeviceCapabilitiesQuery * to
>> AVFormatContext and allocate it on first use and free when context is freed
>> (as you pointed above).
>
> agree, that seems like a good idea
> also the struct can be allocated when the context is
>
> also the structure could be made accessible through
> AVClass.child_next / child_class_next
> from the devices context

This is not helping with anything. You cannot make aliases across 
structures (right?) for input devices, and it shows dummy options for 
output devices. I think this AVDeviceCapabilitesQuery object should be 
used only to replace AVFormatContext object when querying caps.