[FFmpeg-devel] [PATCH] avcodec/decode: guard against NULL hw_frames_ctx

[Rogozhkin, Dmitry V]

On Tue, 2023-11-14 at 19:32 +0100, Hendrik Leppkes wrote:
> On Tue, Nov 14, 2023 at 7:23 PM Dmitry Rogozhkin
> <dmitry.v.rogozhkin-at-intel.com at ffmpeg.org> wrote:
> > 
> > Gurd against segfault running VLC decoding under msys2 [1]:
> > 
> > Thread 33 received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 37728.0xadd0]
> > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00,
> > hwaccel_picture_private=0x65dfd00)
> >     at libavcodec/decode.c:1848
> > 1848        frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx-
> > >data;
> > (gdb) bt
> >     at libavcodec/decode.c:1848
> >     at libavcodec/h264_slice.c:208
> >     first_slice=1) at libavcodec/h264_slice.c:1599
> >     at libavcodec/h264_slice.c:2130
> >     at libavcodec/h264dec.c:652
> >     got_frame=0x646e4b0, avpkt=0x64522c0) at
> > libavcodec/h264dec.c:1048
> > 
> > (gdb) p avctx
> > $1 = (AVCodecContext *) 0x6447b00
> > (gdb) p avctx->hw_frames_ctx
> > $2 = (AVBufferRef *) 0x0
> > 
> > See[1]: https://github.com/msys2/MINGW-packages/pull/19050
> > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv
> > callback")
> > CC: Lynne <dev at lynne.ee>
> > CC: Christoph Reiter <reiter.christoph at gmail.com>
> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin at intel.com>
> > ---
> >  libavcodec/decode.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/libavcodec/decode.c b/libavcodec/decode.c
> > index ad39021..3caaeec 100644
> > --- a/libavcodec/decode.c
> > +++ b/libavcodec/decode.c
> > @@ -1845,9 +1845,9 @@ int
> > ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void
> > **hwaccel_picture_pr
> > 
> >      av_assert0(!*hwaccel_picture_private);
> > 
> > -    frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data;
> > +    frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx ?
> > avctx->hw_frames_ctx->data : NULL;
> >      *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel-
> > >frame_priv_data_size, 0,
> > -                                                      frames_ctx-
> > >device_ctx,
> > +                                                      frames_ctx ?
> > frames_ctx->device_ctx : NULL,
> >                                                        hwaccel-
> > >free_frame_priv);
> 
> The free_frame_priv callback is not usable when no context is
> available. The code should be updated to check if the hwaccel has a
> free_frame_priv callback and then require a frames_ctx to be set and
> otherwise error out (instead of crashing), and if free_frame_priv is
> not set, then it can allocate a buffer without these requirements.

Thank you for feedback. I hope I have fixed this and sent v2 update
yesterday. Can you, please, help review again?

> 
> - Hendrik
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".