[MEncoder-users] MEncoder segfault

[Dieter]

In message <200701171725.38495.microchip at chello.be>, Grozdan Nikolov writes:
> Well, this is what I did:
> 
> after configuring MPlayer, I added the '-g' flag to the 'config.mak' file. I 
> first tried to compile MPlayer with the '--enable-debug' option but it fails, 
> so I only used '-g'
> 
> Then after installation, I ran it under gdb and it segfaultet as expected
> 
> gdb /usr/local/bin/mencoder
> (gdb) run <mencoder_parameters>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1216477488 (LWP 20801)]
> 0x081ec674 in sws_freeFilter ()
> (gdb) bt
> #0  0x081ec674 in sws_freeFilter ()
> #1  0x00016800 in ?? ()
> #2  0xa0001400 in ?? ()
> #3  0x00000000 in ?? ()
> 
> since it always segfaults at sws_freeFilter, I did a 'grep -R sws_freeFilter 
> *' on the MPlayer source code, and those were the files that came up with 
> sws_freeFilter
> 
> libmpcodecs/vf_scale.c: if(src_filter) sws_freeFilter(src_filter);
> libswscale/swscale.c:void sws_freeFilter(SwsFilter *filter){
> libswscale/swscale.h:void sws_freeFilter(SwsFilter *filter);

Gdb should be providing more information than that.  I suspect that swscale.c
was not compiled with -g.

Try:

	make clean
	make

This should recompile with -g.

Then try running mencoder under gdb again.

Here is an example of what gdb should be providing.  (This example is a bug in ffmpeg
rather than mencoder, but you get the idea.)

Program terminated with signal 11, Segmentation fault.
#0  0x0000000000693a0e in L1 ()
[New LWP 100108]
(gdb) bt
#0  0x0000000000693a0e in L1 ()
#1  0x000000000051b1b0 in ff_er_add_slice (s=0xb9a010, startx=0, starty=0, endx=-36259729, endy=-1269859056, status=14)
    at /rw/src/ffmpeg/libavcodec/error_resilience.c:649
#2  0x0000000000507fd9 in mpeg_decode_frame (avctx=0x8dd010, data=0x7fffffffe190, data_size=0x7fffffffdf98, 
    buf=0x210e010 "", buf_size=13169) at /rw/src/ffmpeg/libavcodec/mpeg12.c:3182
#3  0x000000000044b7ea in avcodec_decode_video (avctx=0x8dd010, picture=0x7fffffffe190, got_picture_ptr=0x7fffffffdf98, 
    buf=0x210e010 "", buf_size=13169) at /rw/src/ffmpeg/libavcodec/utils.c:944
#4  0x0000000000402aca in output_packet (ist=0x8c6c90, ist_index=2, ost_table=0x8c3190, nb_ostreams=2, pkt=0x7fffffffe490)
    at /rw/src/ffmpeg/ffmpeg.c:1283
#5  0x0000000000408c2c in main (argc=1245280544, argv=0x8c6c90) at /rw/src/ffmpeg/ffmpeg.c:2121

Note that gdb provides:

	the name of the C function
	the arguments that the function was called with
	the filename that contains the source code for the function
	the line number within the file

In the example, main() called output_packet() which called avcodec_decode_video()
which called mpeg_decode_frame() which called ff_er_add_slice().  Ff_er_add_slice()
called memset() with bad arguments.  Memset() is a library function and the
library wasn't compiled with -g.