[Libav-user] Segmentation faul in flvenc.c

[Mart van de Sanden]

Hi all,

I'm using libavcodec and libavformat for h264 encoding and it seems that 
I might have stumbled upon a bug in libavformat. I was hoping to get 
your opinion about this.

It is illustrated by the gdb session below. the segmentation fault is 
cause by the fact that "enc->extradata == 0x0" while 
"enc->extradata_size = 35". It could potentially be fixed by adding an 
extra check "&& (enc->extradata != NULL)" but I suspect that the bug 
actually occurs at the place where extradata_size is set to 35 while 
extradata receives a null-pointer. I could (and might) go looking for 
this place, but I was hoping that someone here could give me some 
"pointers" :).

Just in case, I apologize if I'm being stupid ;).

cheers,

Mart van de Sanden

--- Start gdb + some stdout log ---

[libx264 @ 0x9cee7940] frame I:1     Avg QP:28.48  size: 30386
[libx264 @ 0x9cee7940] frame P:67    Avg QP:40.20  size:  7642
[libx264 @ 0x9cee7940] mb I  I16..4: 85.9%  0.0% 14.1%
[libx264 @ 0x9cee7940] mb P  I16..4:  5.9%  0.0%  0.9%  P16..4:  2.8%  
0.3%  0.0%  0.0%  0.0%    skip:89.9%
[libx264 @ 0x9cee7940] final ratefactor: 33.55
[libx264 @ 0x9cee7940] coded y,uvDC,uvAC intra: 33.2% 42.8% 29.7% inter: 
1.1% 1.4% 0.4%
[libx264 @ 0x9cee7940] i16 v,h,dc,p: 41% 26% 20% 14%
[libx264 @ 0x9cee7940] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 32% 26%  2%  
3%  2%  4%  2%  5%
[libx264 @ 0x9cee7940] i8c dc,h,v,p: 59% 21% 13%  6%
[libx264 @ 0x9cee7940] Weighted P-Frames: Y:6.0% UV:3.0%
[libx264 @ 0x9cee7940] kb/s:957.12

Program received signal SIGSEGV, Segmentation fault.
0x029f9ce8 in flv_write_packet (s=0x7f7f3da0, pkt=0xbffff038) at 
libavformat/flvenc.c:401
401            if (enc->extradata_size > 0 && *(uint8_t*)enc->extradata 
!= 1) {
(gdb) bt
#0  0x029f9ce8 in flv_write_packet (s=0x7f7f3da0, pkt=0xbffff038) at 
libavformat/flvenc.c:401
#1  0x02a7e71f in av_write_trailer (s=0x7f7f3da0) at 
libavformat/utils.c:3188

... (Deleted a few lines)

#18 0x08050ce7 in main (argc=1, argv=0xbffff404) at 
/home/mart/Projects/yb/test_framework2.cc:34
(gdb) select 0
(gdb) print enc->extradata_size
$1 = 35
(gdb) print enc->extradata
$2 = (uint8_t *) 0x0
(gdb)

--- End log ---

-- 
Mart van de Sanden | yellowBird

yellowBird - See the world like never before
Brugstraat 32 / 9711 HZ Groningen / The Netherlands

M. (Mart) van de Sanden / Software Specialist

T +31 50 8222 822
F +31 50 8501 351
E  mart.vandesanden at yb.nl
W http://www.yb.nl