[Libav-user] customize d3d11va decoder

[Mark Thompson]

On 14/12/2018 21:31, Benjamin de la Fuente Ranea wrote:
> Hello,
> 
> I'm using ffmpeg to decode mp4 videos in hwaccel mode with d3d11va and I'm trying to draw the decoded surface (NV12 format) to the screen using DX11, without copying the surface to CPU and upload again to another texture. For this I need to add D3D11_BIND_SHADER_RESOURCE to hw_frames_ctx->hwctx->BindFlags. These flags are used to create the internal Texture2DArray with all the surfaces needed to decode by ffmpeg, but I can't find a way to do it without modifying ffmpeg source code.
> 
> The way I modify ffmpeg is removing the const qualifier to the declaration and definition of the ff_hwcontext_type_d3d11va struct in hwcontext_d3d11va.c and hwcontext_internal.h, so in my user code I can change all the hooks, in particular "frames_init" function pointer to a custom function that add the D3D11_BIND_SHADER_RESOURCE to the BindFlags and then calls the original d3d11va_frames_init function.
> 
> I'm not sure if there is any other better way to do this. Does anyone knows something that can be useful?

Presumably you are currently using the decoder by setting only the hw_device_ctx field in AVCodecContext to a suitable device, and letting it do allocation internally?

If you need to allocate with some non-default parameters then you want to set the hw_frames_ctx field instead.

If you only want to set the BindFlags, this is provided in the frames context creation through the field you note above in AVD3D11VAFramesContext.  Generally the way to do this is to create a new hwframe context with the desired parameters (including the hwctx structure with your BindFlags) inside the get_format callback, then setting that as hw_frames_ctx for the decoder to use.

That still makes the array texture internally with the flags you specify - if you want even more control, you can make the array texture yourself with arbitrary parameters and set it in hwctx so that it gets wrapped by the frames context.  (Though note that some parameters can make the textures unusable for decoding on particular hardware, so care is needed in this setup.)

- Mark