[FFmpeg-devel] [PATCH V3] avcodec/h264_mp4toannexb_bsf: extract extradata for first coming frame

Mark Thompson sw at jkqxz.net
Sat Sep 29 11:34:54 EEST 2018


On 29/09/18 08:43, Linjie Fu wrote:
> Add "new_nal_slice" to indicate the first coming nal_slice (including
> H264_NAL_IDR_SLICE and H264_NAL_SLICE)
> 
> Extract extradata of streams when the first nal_slice comes, not only
> H264_NAL_IDR_SLICE but also H264_NAL_SLICE.
> 
> This patch aims at the following issues:
> 1. the IDR frame is missing in the  first GOP of a stream
> (common in live stream, IDR.No.Inband.SPPS.mkv in ticket #6418)
> 2. there is no IDR frame in the input stream.
> (No.Inband.SPPS.No.IDR.mkv in ticket #6418)
> 
> Both clips could be decoded successfully using software decoding
> but had problems in hwaccel using qsv before applying the patch.
> 
> V3: modified the commit message and the code duplication.
> 
> Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> ---
>  libavcodec/h264_mp4toannexb_bsf.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/h264_mp4toannexb_bsf.c b/libavcodec/h264_mp4toannexb_bsf.c
> index fb3f24ea40..8dd9159b22 100644
> --- a/libavcodec/h264_mp4toannexb_bsf.c
> +++ b/libavcodec/h264_mp4toannexb_bsf.c
> @@ -33,6 +33,7 @@ typedef struct H264BSFContext {
>      int32_t  pps_offset;
>      uint8_t  length_size;
>      uint8_t  new_idr;
> +    uint8_t  new_nal_slice;
>      uint8_t  idr_sps_seen;
>      uint8_t  idr_pps_seen;
>      int      extradata_parsed;
> @@ -236,13 +237,17 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
>          if (!s->new_idr && unit_type == H264_NAL_IDR_SLICE && (buf[1] & 0x80))
>              s->new_idr = 1;
>  
> -        /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */
> -        if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) {
> +        /* prepend to the first type 5 NAL unit of an IDR picture,
> +         * or the first type 1 NAL unit of an IDR missing picture, 
> +         * if no sps/pps are already present */
> +        if (s->new_idr && !s->idr_sps_seen && !s->idr_pps_seen
> +                    && (unit_type == H264_NAL_IDR_SLICE || (!s->new_nal_slice && H264_NAL_SLICE == unit_type))) {
>              if ((ret=alloc_and_copy(out,
>                                 ctx->par_out->extradata, ctx->par_out->extradata_size,
>                                 buf, nal_size, 1)) < 0)
>                  goto fail;
>              s->new_idr = 0;
> +            s->new_nal_slice = 1;
>          /* if only SPS has been seen, also insert PPS */
>          } else if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && s->idr_sps_seen && !s->idr_pps_seen) {
>              if (s->pps_offset == -1) {
> 

I'm not really against this change, but it's hard to be sure it doesn't perturb something else because the h264_mp4toannexb BSF gets used in a lot of places.  It would seem safer to only do this on real recovery points, rather than blindly splicing it in front of any frame?  The frame you actually put it with here is still unlikely to be usefully decoable if you cut a live stream at a random point.

As an alternative for your libmfx case, most of the other decoders using h264_mp4toannexb (CrystalHD, CUVID, Mediacodec, RKMPP; not V4L2M2M) consume the extradata separately rather than requiring it to be in the stream - if libmfx could do that then it would already be available and there wouldn't be a need to splice it into the stream in a questionable place like this.

(Or, since the normal decoder is fine, just use the appropriate hwaccel (+ hwmap for libmfx output if wanted) rather than the legacy libmfx decoders.)

Thanks,

- Mark


More information about the ffmpeg-devel mailing list