[FFmpeg-devel] [PATCH v2] fftools/ffmpeg: accelerate seeking while reading input at native frame rate

[Linjie Fu]

On Sun, Jul 18, 2021 at 1:21 PM Gyan Doshi <ffmpeg at gyani.pro> wrote:
>
>
>
> On 2021-07-18 10:42, Linjie Fu wrote:
> > Hi Gyan,
> > On Sun, Jul 18, 2021 at 12:24 PM Gyan Doshi <ffmpeg at gyani.pro> wrote:
> >>
> >>
> >> On 2021-07-18 09:32, Linjie Fu wrote:
> >>> On Wed, Jul 7, 2021 at 9:42 AM Linjie Fu <linjie.justin.fu at gmail.com> wrote:
> >>>> On Sun, Jul 4, 2021 at 10:50 PM Linjie Fu <fulinjie at zju.edu.cn> wrote:
> >>>>> From: Linjie Fu <linjie.justin.fu at gmail.com>
> >>>>>
> >>>>> Skip the logic of frame rate emulation until the input reaches the
> >>>>> specified start time.
> >>>>>
> >>>>> Test CMD:
> >>>>>      $ffmpeg -re -ss 30 -i input.mp4 -pix_fmt yuv420p -f sdl2 -
> >>>>>
> >>>>> Before the patch:
> >>>>> first time to got frame, it takes 257305 us
> >>>>> After this patch:
> >>>>> first time to got frame, it takes 48879 us
> >>>>>
> >>>>> Signed-off-by: Linjie Fu <linjie.justin.fu at gmail.com>
> >>>>> ---
> >>>>> [v2]: fixed the mixed declaration and code warning
> >>>>> Calculate the time to get the first frame:
> >>>>> https://github.com/fulinjie/ffmpeg/commit/2aa4762e1e65709997b1ab9dd596332244db80ed
> >>>>>    fftools/ffmpeg.c | 8 ++++++--
> >>>>>    1 file changed, 6 insertions(+), 2 deletions(-)
> >>>>>
> >>>>> diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
> >>>>> index e97d879cb3..c8849e4250 100644
> >>>>> --- a/fftools/ffmpeg.c
> >>>>> +++ b/fftools/ffmpeg.c
> >>>>> @@ -4221,10 +4221,14 @@ static int get_input_packet(InputFile *f, AVPacket **pkt)
> >>>>>    {
> >>>>>        if (f->rate_emu) {
> >>>>>            int i;
> >>>>> +        int64_t pts;
> >>>>> +        int64_t now;
> >>>>>            for (i = 0; i < f->nb_streams; i++) {
> >>>>>                InputStream *ist = input_streams[f->ist_index + i];
> >>>>> -            int64_t pts = av_rescale(ist->dts, 1000000, AV_TIME_BASE);
> >>>>> -            int64_t now = av_gettime_relative() - ist->start;
> >>>>> +            if (!ist->got_output)
> >>>>> +                continue;
> >>>>> +            pts = av_rescale(ist->dts, 1000000, AV_TIME_BASE);
> >>>>> +            now = av_gettime_relative() - ist->start;
> >>>>>                if (pts > now)
> >>>>>                    return AVERROR(EAGAIN);
> >>>>>            }
> >>>>> --
> >>>>> 2.31.1
> >>>> ping, thx.
> >>>>
> >>> Another ping, thx.
> >> I pushed changes to this code yesterday. I don't think it's required
> >> anymore, but do test.
> >>
> > Thanks for the review, tested after applying the readrate patch, I'm
> > afraid that it's not identical as hope,
> > since ist->nb_packets would increase no matter input stream got output or not:
> >
> > (lldb) p ist->nb_packets
> > (uint64_t) $4 = 1
> >
> > (lldb) p ist->got_output
> > (int) $5 = 0
> >
> > Hence we still need to add the check for  ist->got_output, or replace
> > the ist->nb_packets.
>
> No, test the speed, not the parity of got_output. got_output is only
> incremented when the stream is decoded.

The time interval consumed to get the first frame is tested as well,
it's not speeded up:
First time to get the frame, it takes 257395 us.
After adding the check for got_output in decoding scenery, this would
reduce to 48792 us.

Hence it doesn't reduce the latency under decode scenery.

> Won't work with streamcopy.

Got your point.

> nb_packets is the correct check since it's incremented after the initial
> packet is demuxed.

IIRC, the initial packet is demuxed right after the first time we call
get_input_packet()-> av_read_frame(),  and nb_packets would increase.
Hence, the check for ist->nb_packets would reduce the latency for
demuxing the first packet,
and won't help much if we use start_time like "-ss 30". Under the
decoding scenery, we still
need to wait until the pkt->pts specified time then demux the packet.

If no objections, consider to change the check to:

if (!ist->nb_packets || (!ist->got_output && ist->decoding_needed)) continue;

to reduce the waiting time for:
1. demuxing the first packet (if any)
2. demuxing the packet before we got the first decoded output.

> > Also there is a new warning caught by the check in patchwork, probably
> > "mixed declaration and code warning".
> > Will send patches to rebase and fix.
>
> There is already patch for the mixed warning.
Saw it, thx.

- linjie