[FFmpeg-devel] [PATCH] frame: add a time_base field

[Nicolas George]

Marton Balint (12021-09-08):
> So how this is going to work? Will e.g. avcodec_send_frame check the time
> base of the frame and recalculate pts/duration to the encoder time base?
> Same goes for every function which is receiving frames?


Thanks for raising this question.

If we add this field without being very clear on its semantic, then
different parts of the code and applications will quickly start making
slightly different assumptions on it, and the result will be more
complexity but little benefit since we cannot assume the code does the
right thing. And we will have to support it until we decide on an API
break.

Among the question that need answering before considering adding it:

- Is it input or output or both? Is it used by libav* to tell the
  application what time base is used for a frame it produces (output),
  as a reminder and clarification? Or is it also used for the
  application to tell libav* the time base it wants to use (input)?

- If it is input, then is libav* forced to accept it, any value? Or can
  it reject it? Alter it? We have this in libavformat: the application
  can set any time base it wants for AVStream before writing the header,
  but write_header can say "I'm Afraid I Can't Do That, Dave" and choose
  a time base supported by the format.

- If it is input, when is it allowed to change? And again, what happens
  if the change is not supported?

- Who is responsible for keeping it up to date? We have part of the code
  that do:

  frame->pts = av_rescale_q(frame->pts, in->time_base, out->time_base);

  and obviously they do not bother to change frame->time_base, because
  there is no frame->time_base now. Some of the changes can be done
  automatically, but not all. Who will go over the code to check? It
  seems to me we will find bugs where frame->time_base is not updated in
  the next ten years at least.

All in all, this addition seems to make everything more complex. It
really means that all timestamps become full-fledged rationals.
Everywhere we used to store a timestamp we will have to add a time_base
field. Everywhere we used to just compare timestamps with < we will have
to use a rational comparison function.

It is doable, but it is a significant cost. So before considering it, on
top of clarifying the semantic, we also need an answer to the question:
What problem is this helping to solve? And: is there no simpler
solution?


As for a duration field, it is not as bad as a time_base field because
it does not make everything more complex, but it still needs much
clarification on its semantic:

- Is it input, output or both?

- If it is input, what happens if it is not consistent with the
  timestamp of the next frame?

- What happens if it is not consistent with the sample count?

Speaking as the maintainer of libavfilter, and without closing any
further discussion, I say: the duration of a frame is currently defined
as the difference with the timestamp of the next frame, and it should
continue to be. If a duration field is added to the frame, it will only
be used by the framework and/or select few utility filters to fill in
information that may be missing.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210909/05fa5c8f/attachment.sig>