[Libav-user] Possible bug in msmpeg4.c

[Tom Vercauteren]

Hi Folks,

First of all, thanks for the nice piece of software that has been
serving me quite nicely. I have read that bug reports should be made
with the bug tracker but I first wanted to confirm an issue that I am
encountering. I am indeed not really familiar with the ffmpeg code.

I am calling ffmpeg from c++ code to encode MSMPEG4V3 video from
grabbed images. My program segfaulted within get_rl_index. This was
caused because of the a previous issue in ff_msmpeg4_encode_block.

---------------
void ff_msmpeg4_encode_block(MpegEncContext * s, DCTELEM * block, int n)
{
    int level, run, last, i, j, last_index;
    int last_non_zero, sign, slevel;
    int code, run_diff, dc_pred_dir;
    const RLTable *rl;
    const uint8_t *scantable;

    if (s->mb_intra) {
        msmpeg4_encode_dc(s, block[0], n, &dc_pred_dir);
        i = 1;
        if (n < 4) {
               rl = &rl_table[s->rl_table_index];
----------------
The problem arises here as s->rl_table_index is equal to -1. I
attempted a very quick and dirty workaround without actually trying to
understand the code. It is shown below. It seems to actually provide
usable results but, as I said, I have no clue whether this really
makes sense.

----------------
void ff_msmpeg4_encode_block(MpegEncContext * s, DCTELEM * block, int n)
{
    int level, run, last, i, j, last_index;
    int last_non_zero, sign, slevel;
    int code, run_diff, dc_pred_dir;
    const RLTable *rl;
    const uint8_t *scantable;

    if (s->mb_intra) {
        msmpeg4_encode_dc(s, block[0], n, &dc_pred_dir);
        i = 1;
        if (n < 4) {
            if( !( s->rl_table_index>0 && s->rl_table_index<NB_RL_TABLES ) ){
               rl = &rl_table[1];
            }
            else {
               rl = &rl_table[s->rl_table_index];
            }
----------------

Can this problem come from my bad usage of ffmpeg or is this an actual
bug that I should file? Also, would someone suggest a better
workaround? I know that this is not a self-contained test case but I
hope that it provides sufficient information to help.

Best regards,
Tom