[Libav-user] Syncing a seperate audio and video stream

Wed Mar 21 17:47:00 CET 2012

Hi,

Thanks for your reply!

Nicolas George wrote:
> Le primidi 1er germinal, an CCXX, Ludwig Ertl a écrit :
>> I want to use ffmpeg to decode Audio and video streams by a
>> Trendnet TV-IP512P IP Camera in order to feed it to ffserver for
>> streaming. I found a document on the Internet which is describing
...
>> variable or some ugly hack like that), I don't have a clue how to
>> sync them. I suspect that it may have something to do with PTS and
>> DTS timestamps, but I don't know how they are used for audio and
>> video sync in seperate streams.
> The PTS of a frame is the timestamp at which this frame
> should start to be
> played or displayed. That is exactly what you need to sync
> audio and video.
I don't know, however, how they are related to each other in seperate
streams that need to sync.
To be able to test different PTS-Configurations, I updated my sources
(see attachment) so that the algorithm can be switched via the TIMER
#define
I had the idea that I possibly could compensate the PTS offsets by saving
the first Timestamp (for testing just in seconds) when initially reading
the stream and then use it as a starting offset for the sequence number
(of course multiplying it with the sample rate) so that I get this as a
starting time for the stream.
This can be tested by setting the USE_TIMESTAMP #define in the attached
code. My idea was that as there now is a different starting offset in PTS,
the streams would sync, but this doesn't seem to work either.
I provided a sample so that you can do some testing regarding sync
algorithm, if you want (just captured video and audio via wget):
http://www.csp.at/ACS/ACVS.cgi
http://www.csp.at/ACS/ACAS.cgi

Using TIMER_SRT for Audio Decoder and TIMER_SEQ for Video decoder works
fine, I can adjust the offset manually with the -ss parameter if the stream
is directly from the camera (ffmpeg just drops a few video frames in the
beginning), but that's not the goal as I want them to sync automatically.
The provided samples are useful for testing the USE_TIMESTAMP #define:

./ffmpeg -i ACVS.cgi -i ACAS.cgi -v 4 test.mpg

[acsv @ 0x8de5a20] Estimating duration from bitrate, this may be inaccurate
Seems stream 0 codec frame rate differs from container frame rate: 30000.00
(30000/1) -> 30.00 (30/1)
Input #0, acsv, from 'ACVS.cgi':
  Duration: N/A, start: 1201105593.033333, bitrate: N/A
    Stream #0.0: Video: mpeg4, yuv420p, 640x480 [PAR 1:1 DAR 4:3], 30 fps,
30 tbr, 30 tbn, 30k tbc
[acs @ 0x8e43120] Estimating duration from bitrate, this may be inaccurate
Input #1, acs, from 'ACAS.cgi':
  Duration: 00:00:15.16, start: 1201105595.000000, bitrate: 128 kb/s
    Stream #1.0: Audio: pcm_s16le, 8000 Hz, 1 channels, s16, 128 kb/s

As you can see, start has different values now, but this doesn't seem to
have any effect on the synchronisation of both streams either.

> In libavdevice, a lot of demuxers (ALSA, V4L on certain kernels, JACK)
> already return a Unix timestamps at microsecond precision, so
> this is a good
> choice. Also, the other possibilities you tried all return a
> redundant information.
I know, that's where I have the timestamps-Code mentioned below from.

> With the ffmpeg command-line tool, sync issues of that kind
> can be solved
> with the -async option, but I do not find it very elegant. In
> custom code,
> or possibly in a future version of ffmpeg, other algorithms could be
> considered.
I already played with -async, but it didn't have any effect for me.

>
>> What I have already tried was using the clock as PTS for both audio
>> and video:
>>
>> av_set_pts_info(st, 64, 1, 1000000);  /* 64 bits pts in us */
>> pkt->pts = ac->hdr.ulTimeSec * 1000000LL + ac->hdr.ulTimeUSec;
>
> That looks right (except the camelCase), and the best option.
>
>> But this just resulted in a totally garbled video stream.
>
> This is rather strange. Can you show your command line and
> console output in
> that case, and describe what kind of "garbled" you get?
I set the TIMER #define to TIMER_TIM in order to use this method.

../ffmpeg -i ACVS.cgi -i ACAS.cgi -v 4 test.mpg

[acsv @ 0x95cca20] Estimating duration from bitrate, this may be inaccurate
Input #0, acsv, from 'ACVS.cgi':
  Duration: N/A, start: 1201105593.757814, bitrate: N/A
    Stream #0.0: Video: mpeg4, yuv420p, 640x480 [PAR 1:1 DAR 4:3], 30k tbr,
1000k tbn, 30k tbc
[acs @ 0x96105a0] Estimating duration from bitrate, this may be inaccurate
Input #1, acs, from 'ACAS.cgi':
  Duration: 00:00:15.16, start: 1201105595.773438, bitrate: 128 kb/s
    Stream #1.0: Audio: pcm_s16le, 8000 Hz, 1 channels, s16, 128 kb/s
File 'test.mpg' already exists. Overwrite ? [y/N] y
[buffer @ 0x95fad20] w:640 h:480 pixfmt:yuv420p
[NULL @ 0x95ce340] Requested sampling rate unsupported using closest
supported (16000)
[mpeg @ 0x95cda80] VBV buffer size not set, muxing may fail
Output #0, mpeg, to 'test.mpg':
  Metadata:
    encoder         : Lavf53.3.0
    Stream #0.0: Video: mpeg1video, yuv420p, 640x480 [PAR 1:1 DAR 4:3],
q=2-31, 200 kb/s, 90k tbn, 60 tbc
    Stream #0.1: Audio: mp2, 16000 Hz, 1 channels, s16, 64 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #1.0 -> #0.1
Press ctrl-c to stop encoding
*** 1 dup!
*** 1 dup!
*** 1 dup!
*** 1 dup!
*** 1 dup!
...

So this results in a lot of dups.. With -vsync 2, there are no more dups,
but the garbled video remains.
The problem doesn't seem to be related to the audio track, as it's also
happening when just decoding the video.

>>  * ACS (Advanced ip-Camera Stream) demuxer
>>  * Copyright (c) 2012 DI(FH) Ludwig Ertl / CSP GmbH
> This looks promising. But I believe you may have much less
> work if you try
> to merge both files in a single demuxer that can automatically detect
> whether it is audio or video.
As the code and structs are different for each demuxer and the streams
are also not in a combined format, I thought that it would be cleaner to
seperate them in 2 files.

Regards,
Ludwig
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: acsdec.c
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20120321/ee0ce3a2/attachment.c>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: acsvdec.c
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20120321/ee0ce3a2/attachment-0001.c>