[FFmpeg-devel] [PATCH] avfilter: add v360_vulkan filter

TADANO Tokumei aimingoff at pc.nifty.jp
Fri Jul 5 18:11:04 EEST 2024 

On 2024/07/05 0:03, Paul B Mahol wrote:
> On Thu, Jul 4, 2024 at 4:48 PM TADANO Tokumei <aimingoff at pc.nifty.jp> wrote:
> 
>>
>> On 2024/06/25 22:27, Paul B Mahol wrote:
>>> On Tue, Jun 25, 2024 at 3:17 PM Dennis Mungai <dmngaie at gmail.com> wrote:
>>>
>>>> On Tue, 25 Jun 2024 at 16:07, Paul B Mahol <onemda at gmail.com> wrote:
>>>>
>>>>> On Tue, Jun 25, 2024 at 2:47 PM Dennis Mungai <dmngaie at gmail.com>
>> wrote:
>>>>>
>>>>>> On Sun, 28 Jun 2020 at 16:59, James Almer <jamrial at gmail.com> wrote:
>>>>>>
>>>>>>> On 6/27/2020 7:54 AM, Paul B Mahol wrote:
>>>>>>>> On 6/26/20, James Almer <jamrial at gmail.com> wrote:
>>>>>>>>> On 6/26/2020 4:16 PM, Lynne wrote:
>>>>>>>>>> Jun 26, 2020, 19:05 by jamrial at gmail.com:
>>>>>>>>>>
>>>>>>>>>>> On 6/26/2020 1:58 PM, Paul B Mahol wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 6/26/20, James Almer <jamrial at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 6/26/2020 1:13 PM, Paul B Mahol wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 6/26/20, Paul B Mahol <onemda at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 6/24/20, Paul B Mahol <onemda at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: Paul B Mahol <onemda at gmail.com>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Please test it and reports benchmark results on non toyish
>> GPUs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please, I really need it!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why nobody wants to help?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This makes me really really sad.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can try. Can you give an example command line with a publicly
>>>>>>>>>>>>> available sample?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> You can use any sample, samples are irrelevant. just report resolution
>>>>>>>>>>>> used.
>>>>>>>>>>>> Also compare with CPU solution with just v360 filter.
>>>>>>>>>>>>
>>>>>>>>>>>> For yuv420p inputs:
>>>>>>>>>>>>
>>>>>>>>>>>> ffmpeg -init_hw_device vulkan=vulkan -i INPUT -filter_hw_device vulkan
>>>>>>>>>>>> -vf hwupload,v360_vulkan=fisheye:e:ih_fov=180:iv_fov=180:w=2048:h=1024,hwdownload,format=yuv420p -f null -
>>>>>>>>>>>>
>>>>>>>>>>>> ffmpeg -i INPUT -vf v360=fisheye:e:ih_fov=180:iv_fov=180:w=2048:h=1024 -f null -
>>>>>>>>>>>>
>>>>>>>>>>>> Just make sure that you get same output resolution in both cases. If
>>>>>>>>>>>> not, adjust w/h accordingly.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm getting a segfault on vkGetInstanceProcAddr, and i don't know if
>>>>>>>>>>> it's your filter or some issue in the driver. gdb is not being helpful.
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> ffmpeg-devel mailing list
>>>>>>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>>>>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>>>>> ffmpeg-devel-request at ffmpeg.org>  with subject "unsubscribe".
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you test again but replace EXT_EXTERNAL_HOST_MEMORY in
>>>>>>>>>> optional_device_exts
>>>>>>>>>> with EXT_NO_FLAG?
>>>>>>>>>
>>>>>>>>> Applying
>>>>>>>>>
>>>>>>>>>> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
>>>>>>>>>> index 5e51d0390f..49fb7827ba 100644
>>>>>>>>>> --- a/libavutil/hwcontext_vulkan.c
>>>>>>>>>> +++ b/libavutil/hwcontext_vulkan.c
>>>>>>>>>> @@ -228,7 +228,7 @@ static const VulkanOptExtension optional_device_exts[]
>>>>>>>>>> = {
>>>>>>>>>>        { VK_EXT_EXTERNAL_MEMORY_DMA_BUF_EXTENSION_NAME,
>>>>>>>>>> EXT_EXTERNAL_DMABUF_MEMORY, },
>>>>>>>>>>        { VK_EXT_IMAGE_DRM_FORMAT_MODIFIER_EXTENSION_NAME,
>>>>>>>>>> EXT_DRM_MODIFIER_FLAGS,     },
>>>>>>>>>>        { VK_KHR_EXTERNAL_SEMAPHORE_FD_EXTENSION_NAME,
>>>>>>>>>> EXT_EXTERNAL_FD_SEM,        },
>>>>>>>>>> -    { VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME,
>>>>>>>>>> EXT_EXTERNAL_HOST_MEMORY,   },
>>>>>>>>>> +    { VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME,
>>>>>>>>>> EXT_NO_FLAG,   },
>>>>>>>>>>    };
>>>>>>>>>>
>>>>>>>>>>    /* Converts return values to strings */
>>>>>>>>>
>>>>>>>>> I still get a segfault:
>>>>>>>>>
>>>>>>>>>> Thread 1 received signal SIGSEGV, Segmentation fault.
>>>>>>>>>> 0x00007ff8ed9274c0 in vkGetInstanceProcAddr ()
>>>>>>>>>>      from
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>> C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ccad5caddc3a3d35\nvoglv64.dll
>>>>>>>>>
>>>>>>>>> Debug output shows
>>>>>>>>>
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] GPU listing:
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     0: GeForce GTX 1660 (discrete)
>>>>>>>>>> (0x2184)
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 0 (total queues:
>>>>>>>>>> 16) for graphics
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 0 flags: (graphics)
>>>>>>>>>> (compute) (transfers) (sparse)
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 2 (total queues:
>>>>>>>>>> 8) for compute
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 2 flags: (compute)
>>>>>>>>>> (transfers) (sparse)
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 1 (total queues:
>>>>>>>>>> 2) for transfers
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 1 flags: (transfers)
>>>>>>>>>> (sparse)
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device extension
>>>>>>>>>> "VK_EXT_external_memory_host"
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device extension
>>>>>>>>>> VK_EXT_external_memory_host
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device: GeForce GTX 1660
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Alignments:
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]
>>>>>>>>>> optimalBufferCopyRowPitchAlignment: 1
>>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]   minMemoryMapAlignment:      64
>>>>>>>>>
>>>>>>>>> And crashes right after printing:
>>>>>>>>>
>>>>>>>>>> [Parsed_v360_vulkan_1 @ 000001532abd5c80] Shader v360_compute linked!
>>>>>>>>>> Size: 5508 bytes
>>>>>>>>>
>>>>>>>>> I'm using the latest driver, 451.48, on Windows 10 2004.
>>>>>>>>
>>>>>>>> So other vulkan filters, like vulkan_chromaber works for you?
>>>>>>>
>>>>>>> All fail the same way in vkGetInstanceProcAddr, so the issue is in
>>>>>>> hwcontext.
>>>>>>>
>>>>>>> Lynne should be able to test/bench this filter.
>>>>>>>
>>>>>>
>>>>>>    Hello Paul, et al,
>>>>>>
>>>>>> Any updates on this filter?
>>>>>>
>>>>>
>>>>> I'm no longer working on FFmpeg.
>>>>>
>>>>> The v360_opencl filter got in more useful state but due complete lack of
>>>>> interest in it from users, I forgot about its existence.
>>>>> The v360_vulkan filter is probably in worse shape than v360_opencl filter
>>>>> and 0/negative interest, and I forgot about its existence too.
>>>>>
>>>>
>>>> Is there an FFmpeg tree with v360_opencl (or a similar filter) out there?
>>>>
>>>
>>>
>>> I can give you patches/files for both v360 vulkan and opencl filter if you
>>> want to hack on it.
>>> But I will not adapt them for free to current library versions, nor provide
>>> any kind of free support for it.
>>>
>>
>> I tested v360_opencl filter with slight modification to your patch.
>> It works, but it generates different output from v360 filter.
>>
>> For example, I tested a sample GoPro Max 360 video on:
>> https://drive.google.com/drive/folders/1DlP6FVw-O6AIm6O0nSsC-mIoCrEBtmdG
>>
>> ffmpeg -i GS010010.360 -init_hw_device opencl:0.0 -vf
>> "hwupload,v360_opencl=yaw=90,hwdownload,format=yuvj420p" output-v360cl.mkv
>> ffmpeg -i GS010010.360 -vf v360=yaw=90 output-v360.mkv
>>
>> I'm not sure which is correct, but I feel the output of v360_opencl filter
>> is better.
>>
> 
> Make sure that difference is not because you used yuvj420p pixel format at
> end.
> The output should be very similar - minus rounding, with visually no
> difference.
> Also why you transcode?

Without specifying format parameter, the ffmpeg command would fails:

$ ffmpeg -i GS010010.360 -init_hw_device opencl:0.0 -vf "hwupload,v360_opencl=yaw=90,hwdownload" output-v360cl.mkv
...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GS010010.360':
...
    Stream #0:0[0x1](eng): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, bt709), 4096x1344 [SAR 1:1 DAR 64:21], 29970 kb/s, 29.97 fps, 29.97 tbr, 90k tbn (default)
...
[hwdownload @ 0x7ef9e4005680] Invalid output format gray for hwframe download.
[Parsed_hwdownload_2 @ 0x7ef9e4005540] Failed to configure output pad on Parsed_hwdownload_2
[vf#0:0 @ 0x5621681baf00] Error reinitializing filters!
[vf#0:0 @ 0x5621681baf00] Task finished with error code: -22 (Invalid argument)
[vf#0:0 @ 0x5621681baf00] Terminating thread with return code -22 (Invalid argument)
[vost#0:0/libx264 @ 0x5621680cf680] Could not open encoder before EOF
[vost#0:0/libx264 @ 0x5621680cf680] Task finished with error code: -22 (Invalid argument)
[vost#0:0/libx264 @ 0x5621680cf680] Terminating thread with return code -22 (Invalid argument)
[out#0/matroska @ 0x562167eac000] Nothing was written into output file, because at least one of its streams received no packets.

The yuvj420p is same as input.
I didn't intended to transcode.

I'm not sure if this ML accepts attachments, but I attach (resized) snapshots.

> 
>>
>>>
>>>> A GPU-accelerated v360 filter is definitely useful to test and
>> implement.
>>>> Definitely more useful than the DNN stack in FFmpeg (by leaps and
>> bounds)
>>>> as far as applicability is concerned.
>>>> _______________________________________________
>>>> ffmpeg-devel mailing list
>>>> ffmpeg-devel at ffmpeg.org
>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>
>>>> To unsubscribe, visit link above, or email
>>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>>>>
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel at ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
-------------- next part --------------
A non-text attachment was scrubbed...
Name: v360.png
Type: image/png
Size: 200215 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20240706/ac58531a/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: v360_opencl.png
Type: image/png
Size: 140112 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20240706/ac58531a/attachment-0001.png>

More information about the ffmpeg-devel mailing list