[FFmpeg-devel] [PATCH 2/3] avcodec/cbs_h266_syntax_template: sanity check num_multi_layer_olss

[James Almer]

On 1/26/2024 6:46 PM, Michael Niedermayer wrote:
> It is not possible to encode a index into an empty list. Thus
> this must be invalid at this point or before.
> Its likely a broader earlier check can be used here, someone knowing
> VVC should look at that. Its not immedeatly obvious from the spec
> by looking for numlayerolss

Can you check if the following fixes it?

> diff --git a/libavcodec/cbs_h266_syntax_template.c b/libavcodec/cbs_h266_syntax_template.c
> index 549d021211..40572dadb5 100644
> --- a/libavcodec/cbs_h266_syntax_template.c
> +++ b/libavcodec/cbs_h266_syntax_template.c
> @@ -793,6 +793,7 @@ static int FUNC(vps) (CodedBitstreamContext *ctx, RWContext *rw,
>      {
>          //calc NumMultiLayerOlss
>          int m;
> +        int num_layers_in_ols = 0;
>          uint8_t dependency_flag[VVC_MAX_LAYERS][VVC_MAX_LAYERS];
>          uint16_t num_output_layers_in_ols[VVC_MAX_TOTAL_NUM_OLSS];
>          uint8_t num_sub_layers_in_layer_in_ols[VVC_MAX_TOTAL_NUM_OLSS][VVC_MAX_TOTAL_NUM_OLSS];
> @@ -895,7 +896,6 @@ static int FUNC(vps) (CodedBitstreamContext *ctx, RWContext *rw,
>                  return AVERROR_INVALIDDATA;
>          }
>          for (i = 1; i < total_num_olss; i++) {
> -            int num_layers_in_ols = 0;
>              if (current->vps_each_layer_is_an_ols_flag) {
>                  num_layers_in_ols = 1;
>              } else if (current->vps_ols_mode_idc == 0 ||

num_layers_in_ols is not meant to be reset on every loop.