[FFmpeg-devel] [PATCH] avformat/dv: fix timestamps of audio packets in case of dropped corrupt audio frames
Dave Rice
dave at dericed.com
Sat Oct 31 22:32:53 EET 2020
> On Oct 31, 2020, at 3:47 PM, Marton Balint <cus at passwd.hu> wrote:
> On Sat, 31 Oct 2020, Dave Rice wrote:
>
>> Hi Marton,
>>
>>> On Oct 31, 2020, at 12:56 PM, Marton Balint <cus at passwd.hu> wrote:
>>> Fixes out of sync timestamps in ticket #8762.
>>
>> Although Michael’s recent patch does address the issue documented in 8762, I haven’t found this patch to fix the issue. I tried with -c:a copy and with -c:a pcm_s16le with some sample files that exhibit this issue but each output was out of sync. I put an output at https://gist.github.com/dericed/659bd843bd38b6f24a60198b5e345795. That output notes that 3597 packages of video are read and 3586 packets of audio. In the resulting file, at the end of the timeline the audio is 9 frames out of sync and my output video stream is 00:02:00.020 and output audio stream is 00:01:59.653.
>>
>> Beyond copying or encoding the audio, are there other options I should use to test this?
>
> Well, it depends on what you want. After this patch you should get a file which has audio packets synced to video, but the audio stream is sparse, not every video packet has a corresponding audio packet. (It looks like our MOV muxer does not support muxing of sparse audio therefore does not produce proper timestamps. But MKV does, please try that.)
>
> You can also make ffmpeg generate the missing audio based on packet timestamps. Swresample has an async=1 option, so something like this should get you synced audio with continous audio packets:
> ffmpeg -y -i 1670520000_12.dv -c:v copy \
> -af aresample=async=1:min_hard_comp=0.01 -c:a pcm_s16le 1670520000_12.mov
Thank you for this. With the patch and async, the result is synced and the resulting audio was the same as Michael’s patch.
Could you explain why you used min_hard_comp here? IIUC min_hard_comp is a set a threshold between the strategies of trim/fill or stretch/squeeze to align the audio to time; however, the async documentation says "Setting this to 1 will enable filling and trimming, larger values represent the maximum amount in samples that the data may be stretched or squeezed” so I thought that async=1 would not permit stretch/squeeze anyway.
> Regards,
> Marton
>
>
>>
>>> Signed-off-by: Marton Balint <cus at passwd.hu>
>>> ---
>>> libavformat/dv.c | 16 ++--------------
>>> tests/ref/seek/lavf-dv | 18 +++++++++---------
>>> 2 files changed, 11 insertions(+), 23 deletions(-)
>>> diff --git a/libavformat/dv.c b/libavformat/dv.c
>>> index 3e0d12c0e3..26a78139f5 100644
>>> --- a/libavformat/dv.c
>>> +++ b/libavformat/dv.c
>>> @@ -49,7 +49,6 @@ struct DVDemuxContext {
>>> uint8_t audio_buf[4][8192];
>>> int ach;
>>> int frames;
>>> - uint64_t abytes;
>>> };
>>> static inline uint16_t dv_audio_12to16(uint16_t sample)
>>> @@ -258,7 +257,7 @@ static int dv_extract_audio_info(DVDemuxContext *c, const uint8_t *frame)
>>> c->ast[i] = avformat_new_stream(c->fctx, NULL);
>>> if (!c->ast[i])
>>> break;
>>> - avpriv_set_pts_info(c->ast[i], 64, 1, 30000);
>>> + avpriv_set_pts_info(c->ast[i], 64, c->sys->time_base.num, c->sys->time_base.den);
>>> c->ast[i]->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
>>> c->ast[i]->codecpar->codec_id = AV_CODEC_ID_PCM_S16LE;
>>> @@ -387,8 +386,7 @@ int avpriv_dv_produce_packet(DVDemuxContext *c, AVPacket *pkt,
>>> for (i = 0; i < c->ach; i++) {
>>> c->audio_pkt[i].pos = pos;
>>> c->audio_pkt[i].size = size;
>>> - c->audio_pkt[i].pts = c->abytes * 30000 * 8 /
>>> - c->ast[i]->codecpar->bit_rate;
>>> + c->audio_pkt[i].pts = (c->sys->height == 720) ? (c->frames & ~1) : c->frames;
>>> ppcm[i] = c->audio_buf[i];
>>> }
>>> if (c->ach)
>>> @@ -401,10 +399,7 @@ int avpriv_dv_produce_packet(DVDemuxContext *c, AVPacket *pkt,
>>> c->audio_pkt[2].size = c->audio_pkt[3].size = 0;
>>> } else {
>>> c->audio_pkt[0].size = c->audio_pkt[1].size = 0;
>>> - c->abytes += size;
>>> }
>>> - } else {
>>> - c->abytes += size;
>>> }
>>>
>>> /* Now it's time to return video packet */
>>> @@ -444,13 +439,6 @@ static int64_t dv_frame_offset(AVFormatContext *s, DVDemuxContext *c,
>>> void ff_dv_offset_reset(DVDemuxContext *c, int64_t frame_offset)
>>> {
>>> c->frames = frame_offset;
>>> - if (c->ach) {
>>> - if (c->sys) {
>>> - c->abytes = av_rescale_q(c->frames, c->sys->time_base,
>>> - (AVRational) { 8, c->ast[0]->codecpar->bit_rate });
>>> - } else
>>> - av_log(c->fctx, AV_LOG_ERROR, "cannot adjust audio bytes\n");
>>> - }
>>> c->audio_pkt[0].size = c->audio_pkt[1].size = 0;
>>> c->audio_pkt[2].size = c->audio_pkt[3].size = 0;
>>> }
>>> diff --git a/tests/ref/seek/lavf-dv b/tests/ref/seek/lavf-dv
>>> index 0000ff5abe..f63e4460be 100644
>>> --- a/tests/ref/seek/lavf-dv
>>> +++ b/tests/ref/seek/lavf-dv
>>> @@ -7,9 +7,9 @@ ret: 0 st: 0 flags:0 ts: 0.800000
>>> ret: 0 st: 0 flags:1 dts: 0.800000 pts: 0.800000 pos:2880000 size:144000
>>> ret: 0 st: 0 flags:1 ts:-0.320000
>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>> -ret: 0 st: 1 flags:0 ts: 2.576667
>>> +ret: 0 st: 1 flags:0 ts: 2.560000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> -ret: 0 st: 1 flags:1 ts: 1.470833
>>> +ret: 0 st: 1 flags:1 ts: 1.480000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> ret: 0 st:-1 flags:0 ts: 0.365002
>>> ret: 0 st: 0 flags:1 dts: 0.360000 pts: 0.360000 pos:1296000 size:144000
>>> @@ -19,9 +19,9 @@ ret: 0 st: 0 flags:0 ts: 2.160000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> ret: 0 st: 0 flags:1 ts: 1.040000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> -ret: 0 st: 1 flags:0 ts:-0.058333
>>> +ret: 0 st: 1 flags:0 ts:-0.040000
>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>> -ret: 0 st: 1 flags:1 ts: 2.835833
>>> +ret: 0 st: 1 flags:1 ts: 2.840000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> ret: 0 st:-1 flags:0 ts: 1.730004
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> @@ -31,10 +31,10 @@ ret: 0 st: 0 flags:0 ts:-0.480000
>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>> ret: 0 st: 0 flags:1 ts: 2.400000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> -ret: 0 st: 1 flags:0 ts: 1.306667
>>> -ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> -ret: 0 st: 1 flags:1 ts: 0.200833
>>> +ret: 0 st: 1 flags:0 ts: 1.320000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> +ret: 0 st: 1 flags:1 ts: 0.200000
>>> +ret: 0 st: 0 flags:1 dts: 0.200000 pts: 0.200000 pos: 720000 size:144000
>>> ret: 0 st:-1 flags:0 ts:-0.904994
>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>> ret: 0 st:-1 flags:1 ts: 1.989173
>>> @@ -43,9 +43,9 @@ ret: 0 st: 0 flags:0 ts: 0.880000
>>> ret: 0 st: 0 flags:1 dts: 0.880000 pts: 0.880000 pos:3168000 size:144000
>>> ret: 0 st: 0 flags:1 ts:-0.240000
>>> ret: 0 st: 0 flags:1 dts: 0.000000 pts: 0.000000 pos: 0 size:144000
>>> -ret: 0 st: 1 flags:0 ts: 2.671667
>>> +ret: 0 st: 1 flags:0 ts: 2.680000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> -ret: 0 st: 1 flags:1 ts: 1.565833
>>> +ret: 0 st: 1 flags:1 ts: 1.560000
>>> ret: 0 st: 0 flags:1 dts: 0.960000 pts: 0.960000 pos:3456000 size:144000
>>> ret: 0 st:-1 flags:0 ts: 0.460008
>>> ret: 0 st: 0 flags:1 dts: 0.480000 pts: 0.480000 pos:1728000 size:144000
>>> — 2.26.2
>>
>> Best Regards,
>> Dave Rice
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list