[Libav-user] how to handle missing pts values to keep audio sync (mpegts->mp4)

Thu Jul 18 11:09:16 CEST 2013

>

Hello!

This is my first mail on this list. So i hope i didn't got the wrong mailinglist.

Im writing an program which received a mpeg ts udp stream and convert it to mp4 on the fly.

Sometimes i have a problem with async audio.

[mpegts @ 0x35fd580] PES packet size mismatch
[mpeg2video @ 0x3604200] skipped MB in I frame at 25 32
[mpeg2video @ 0x3604200] skipped MB in I frame at 1 33
[mpeg2video @ 0x3604200] ac-tex damaged at 0 34
[mpeg2video @ 0x3604200] ac-tex damaged at 0 35
[mpeg2video @ 0x3604200] Warning MVs not available
[mpeg2video @ 0x3604200] concealing 176 DC, 176 AC, 176 MV errors in I frame
[mp2 @ 0x3649fa0] Header missing
[err]failed to decode audio packet

I use udp so it is possible to loose a packages. I have to handle it.

my video_push and audio_push are appended.

I dump out the PTS before i push it to the encoder.

So you can see that 67 PTS Values are missing.

With a 25fps video it is a delay of 2,68 sec. That's exactly what i see if i play the mp4 file via vlc.
 I have tried to skip some video frames, but din't have any effect.
 How to bring back sync audio?
 skip audio or video frames?
 recalc pts values?
 add dummy frames?
 all together?

push audio frame to 689152 (673)
push audio frame to 690176 (674)
[mpegts @ 0x2163580] PES packet size mismatch
push video frame to 392
[mpeg2video @ 0x216a200] skipped MB in I frame at 25 32
[mpeg2video @ 0x216a200] skipped MB in I frame at 1 33
[mpeg2video @ 0x216a200] ac-tex damaged at 0 34
[mpeg2video @ 0x216a200] ac-tex damaged at 0 35
[mpeg2video @ 0x216a200] Warning MVs not available
[mpeg2video @ 0x216a200] concealing 176 DC, 176 AC, 176 MV errors in I frame
push video frame to 393
push video frame to 394<----------------------------- 1 PTS Values is missing!!!!!!!
push video frame to 396
push video frame to 397
[mp2 @ 0x21affa0] Header missing
[err]failed to decode audio packet
push audio frame to 691200 (675)
push audio frame to 692224 (676)
push audio frame to 693248 (677)
push audio frame to 694272 (678)
push audio frame to 695296 (679)
push audio frame to 696320 (680)
push audio frame to 697344 (681)
push audio frame to 698368 (682)
push audio frame to 699392 (683)
push audio frame to 700416 (684)
push audio frame to 701440 (685)
push audio frame to 702464 (686)
push audio frame to 703488 (687)
push audio frame to 704512 (688)
push audio frame to 705536 (689)
push audio frame to 706560 (690)
push audio frame to 707584 (691)
push audio frame to 708608 (692)
push video frame to 398
push video frame to 465                           <----------------------------- 67 PTS Values are missing!!!!!!!
push video frame to 466

my video_push look something like that:

int enc_audio_frame (my_enc_t *enc,AVFrame *audio_frame){

  if ( enc->first_video_pts == AV_NOPTS_VALUE)
        return 0;

  if (enc->first_audio_pts == AV_NOPTS_VALUE) {
        enc->first_audio_pts = audio_frame->pts;
  }

  if (audio_frame->pts != AV_NOPTS_VALUE) {
      audio_frame->pts = av_rescale_q(audio_frame->pts - enc->first_audio_pts, my_decoder_output.a_time_base, enc->conf.a_time_base);
  }
  AVPacket *enc_packet = (AVPacket *) malloc (sizeof (AVPacket));
  av_init_packet(enc_packet);
  enc_packet->size = 0;
  enc_packet->data=NULL;
  int packet_done = 0;
  if ( avcodec_encode_audio2 (enc->audio_codec_ctx,enc_packet,audio_frame,&packet_done) < 0) {
      fprintf (stderr,"[err]%s failed to encode audio samples\n", __FUNCTION__);
      return -1;
  }
  if ( packet_done ) {
    enc_packet->stream_index= 1;
    fprintf (stderr,"push audio frame to %"PRId64" (%"PRId64")\n", audio_frame->pts, audio_frame->pts/1024);
    enc->push(enc->push_priv,enc_packet);
  }
  av_free_packet(enc_packet);
  av_free (enc_packet);

  return 0;

}

my audio_push look something like that:

int enc_audio_frame (my_enc_t *enc,AVFrame *audio_frame){

  if ( enc->first_video_pts == AV_NOPTS_VALUE)
        return 0;

  if (enc->first_audio_pts == AV_NOPTS_VALUE) {
        enc->first_audio_pts = audio_frame->pts;
  }

  if (audio_frame->pts != AV_NOPTS_VALUE) {
      audio_frame->pts = av_rescale_q(audio_frame->pts - enc->first_audio_pts, my_decoder_output.a_time_base, enc->conf.a_time_base);
  }
  AVPacket *enc_packet = (AVPacket *) malloc (sizeof (AVPacket));
  av_init_packet(enc_packet);
  enc_packet->size = 0;
  enc_packet->data=NULL;
  int packet_done = 0;
  if ( avcodec_encode_audio2 (enc->audio_codec_ctx,enc_packet,audio_frame,&packet_done) < 0) {
      fprintf (stderr,"[err]%s failed to encode audio samples\n", __FUNCTION__);
      return -1;
  }
  if ( packet_done ) {
    enc_packet->stream_index= 1;
    fprintf (stderr,"push audio frame to %"PRId64" (%"PRId64")\n", audio_frame->pts, audio_frame->pts/1024);

    enc->push(enc->push_priv,enc_packet);
  }
  av_free_packet(enc_packet);
  av_free (enc_packet);

  return 0;

If i use the application ffmpeg i got also an async mp4:

ffmpeg -i  ~/Videos/defekte_streams/asyncron_1.ts   test.mp4
ffmpeg version 0.8.6-6:0.8.6-0ubuntu0.12.10.1, Copyright (c) 2000-2013 the Libav developers
  built on Apr  2 2013 17:02:16 with gcc 4.7.2
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
[mp3 @ 0xe71700] Header missing
[mpegts @ 0xe6a180] max_analyze_duration reached
[NULL @ 0xe8dbc0] start time is not set in estimate_timings_from_pts
[NULL @ 0xe8fbe0] start time is not set in estimate_timings_from_pts
[mpegts @ 0xe6a180] PES packet size mismatch
Input #0, mpegts, from '/home/patrick/Videos/defekte_streams/asyncron_1.ts':
  Duration: 00:03:53.69, start: 94926.112233, bitrate: 3423 kb/s
  Program 130 
    Stream #0.0[0x1411]: Video: mpeg2video (Main), yuv420p, 704x576 [PAR 16:11 DAR 16:9], 10000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0.1[0x1412](deu): Audio: mp2, 48000 Hz, stereo, s16, 192 kb/s
    Stream #0.2[0x1414](deu): Subtitle: [6][0][0][0] / 0x0006
    Stream #0.3[0x87c]: Data: [11][0][0][0] / 0x000B
    Stream #0.4[0xa6e]: Data: [5][0][0][0] / 0x0005
File 'test.mp4' already exists. Overwrite ? [y/N] y
[buffer @ 0xe9c480] w:704 h:576 pixfmt:yuv420p
[libx264 @ 0xe70560] using SAR=16/11
[libx264 @ 0xe70560] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
[libx264 @ 0xe70560] profile Main, level 3.0
[libx264 @ 0xe70560] 264 - core 123 r2189 35cf912 - H.264/MPEG-4 AVC codec - Copyleft 2003-2012 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=0 b_adapt=1 b_bias=0 direct=1 weightb=0 open_gop=1 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.25 aq=1:1.00
Output #0, mp4, to 'test.mp4':
  Metadata:
    encoder         : Lavf53.21.1
    Stream #0.0: Video: libx264, yuv420p, 704x576 [PAR 16:11 DAR 16:9], q=-1--1, 25 tbn, 25 tbc
    Stream #0.1(deu): Audio: libvo_aacenc, 48000 Hz, stereo, s16, 200 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
Press ctrl-c to stop encoding
[mp2 @ 0xe71700] Header missing
Error while decoding stream #0.1
[mpegts @ 0xe6a180] Continuity check failed for pid 5137 expected 6 got 8/s dup=21 drop=0    
[mpegts @ 0xe6a180] Continuity check failed for pid 18 expected 15 got 7
[mpegts @ 0xe6a180] Continuity check failed for pid 5138 expected 7 got 1
[mpegts @ 0xe6a180] PES packet size mismatch
[mpegts @ 0xe6a180] Continuity check failed for pid 2670 expected 15 got 12
[mpegts @ 0xe6a180] Continuity check failed for pid 2172 expected 6 got 9
[mpegts @ 0xe6a180] Continuity check failed for pid 0 expected 15 got 3
[mpeg2video @ 0xe6f6a0] skipped MB in I frame at 25 32
[mpeg2video @ 0xe6f6a0] skipped MB in I frame at 1 33
[mpeg2video @ 0xe6f6a0] ac-tex damaged at 0 34
[mpeg2video @ 0xe6f6a0] ac-tex damaged at 0 35
[mpeg2video @ 0xe6f6a0] Warning MVs not available
[mpeg2video @ 0xe6f6a0] concealing 176 DC, 176 AC, 176 MV errors
[mp2 @ 0xe71700] Header missing=    1410kB time=14.92 bitrate= 774.4kbits/s dup=21 drop=0    
Error while decoding stream #0.1
[mpegts @ 0xe6a180] PES packet size mismatchime=230.12 bitrate=1401.1kbits/s dup=88 drop=0    
[mpeg2video @ 0xe6f6a0] ac-tex damaged at 36 13
[mpeg2video @ 0xe6f6a0] Warning MVs not available
[mpeg2video @ 0xe6f6a0] concealing 1012 DC, 1012 AC, 1012 MV errors
[mp2 @ 0xe71700] incomplete frame
Error while decoding stream #0.1
frame= 5851 fps= 28 q=28.0 Lsize=   40194kB time=230.66 bitrate=1427.5kbits/s dup=88 drop=0    
video:34406kB audio:5631kB global headers:0kB muxing overhead 0.391637%
frame I:58    Avg QP:21.87  size: 27346
[libx264 @ 0xe70560] frame P:3250  Avg QP:24.10  size:  8797
[libx264 @ 0xe70560] frame B:2543  Avg QP:26.71  size:  1988
[libx264 @ 0xe70560] consecutive B-frames: 31.4% 26.7% 15.4% 26.5%
[libx264 @ 0xe70560] mb I  I16..4: 44.1%  0.0% 55.9%
[libx264 @ 0xe70560] mb P  I16..4: 11.1%  0.0%  5.9%  P16..4: 45.6%  9.3%  4.3%  0.0%  0.0%    skip:23.7%
[libx264 @ 0xe70560] mb B  I16..4:  0.6%  0.0%  0.4%  B16..8: 32.4%  2.8%  0.6%  direct: 1.9%  skip:61.3%  L0:36.1% L1:56.3% BI: 7.6%
[libx264 @ 0xe70560] coded y,uvDC,uvAC intra: 38.9% 54.4% 16.7% inter: 16.6% 20.4% 1.4%
[libx264 @ 0xe70560] i16 v,h,dc,p: 45% 28% 12% 14%
[libx264 @ 0xe70560] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 19% 21%  5%  6%  7%  5%  6%  5%
[libx264 @ 0xe70560] i8c dc,h,v,p: 51% 24% 20%  4%
[libx264 @ 0xe70560] Weighted P-Frames: Y:10.6% UV:6.0%
[libx264 @ 0xe70560] ref P L0: 65.7% 11.6% 15.1%  7.2%  0.4%
[libx264 @ 0xe70560] ref B L0: 84.6% 15.4%
[libx264 @ 0xe70560] kb/s:1204.27

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20130718/c9410d8c/attachment.html>