[FFmpeg-devel] [PATCH 0/2] Implement SMPTE 2038 output support over Decklink SDI

[Marton Balint]

On Mon, 24 Apr 2023, Devin Heitmueller wrote:

> Hello Marton,
>
> Thanks for reviewing.  Comments inline:
>
> On Sun, Apr 23, 2023 at 2:43 PM Marton Balint <cus at passwd.hu> wrote:
>> In general, queueing packets in specific components should be avoided if
>> possible. Muxed packets are normally ordered by DTS and stream id, generic
>> code ensures that. If you want something other than that, then I think
>> the perferred way of doing it is by providing a custom interleave
>> function. (e.g. to ensure you get data packets before video even if data
>> stream has a higher stream ID.)
>
> To be clear, using a queue was not first choice.  It's the result of
> trying different approaches, and I'm open to constructive suggestions
> on alternatives.
>
> While what you're are saying is correct "in general", there are some
> really important reasons why it doesn't work in this case.  Permit me
> to explain...
>
> By default, the behavior of the mux interleaver is to wait until there
> is at least one packet available for each stream before writing to the
> output module (in this case decklink).  However data formats such as
> SMPTE ST2038 are considered to be "sparse" as there isn't necessarily
> a continuous stream of packets like with video and audio (there may be
> many seconds between packets, or no packets at all).  As a result you
> can't wait for a packet to be available on all streams since on some
> streams it will simply wait continuously until hitting the
> max_interleave_delta, at which point it will burst out everything in
> the queue.  This would cause stalls and/or stuttering playback on the
> decklink output.
>
> To accommodate these sparse streams we added code to mux.c to not wait
> for 2038 packets.  A side-effect of that though is that packets will
> be sent through as soon as they hit the mux, which in most cases will
> be significantly ahead of the video (potentially hundreds of
> milliseconds).  This can easily be seen experimentally by adding an
> av_log() line to ff_decklink_write_packet(), which will show in many
> cases the PTS values of the data frames being sent 20+ frames before
> the corresponding video.

Okay, I realized there is one thing here I don't understand. What if we 
interleave data packets the same way as others, but we don't wait for them 
in order to start flushing packet queues?

So I wonder, if you removed the AV_CODEC_ID_SMPTE_2038 exception
from init_muxer when calculating si->nb_interleaved_streams but keep the 
exception in ff_interleave_packet_per_dts, and set 
max_interleave_delta to 1, would that work?

Regards,
Marton