[FFmpeg-devel] Asynchronously delivering data to FFmpeg, and extending the API to do this...

[Nicolas George]

Philip Prindeville (2018-03-19):
> I’m looking at the API and trying to figure out how to adapt it to our
> use, but without much luck.  If I get this working, I’ll try to
> upstream the enhancement in case it’s generically useful.
> 
> The problem in a nutshell is we’re using libevent2 and everything is
> event-driven, including notifications about data arriving from the
> network.
> 
> So rather than using avio_alloc_context() where the read_packet()
> function does a blocking read on a socket, I need to do the inverse:
> call a function which collects data until it’s parsed a complete
> frame, then return an indication of that.
> 
> Maybe something like:
> 
> bool avio_collect_frame(AVIOContext *avio, uint_t *ptr, size_t *size);
> 
> where it’s called with an AVIOContext where data is accumulated
> internally (and parsed), ptr is a pointer to the new data being
> presented, and size is a pointer to the number of bytes available (it
> may get rewritten with the number of bytes actually used… or it could
> just save the overflow from the current frame and use it for the
> subsequent frame).
> 
> Then in the (read) event handler for data arriving on a socket, we pass each new block to data to this function until it returns true.  When it returns true, then we activate an event to decode/render the frame (or handle it however else we might want to).
> 
> i.e. something like:
> 
> struct event *ev_render;
> 
> …
> 
> 
> ev_render = event_new(evbase, -1, 0, render_cb, ctx);
> event_add(ev_render, NULL);
> 
> 
> and to fire the renderer:
> 
> 	bytes = bp->size;
> 	if (avio_collect_frame(avio, bp->base, &bytes))
> 		event_active(ev_render, 0, 1);
> 
> etc.
> 
> What’s involved in doing this?

This is something I have been considering for a long time. It would be
very useful, but unfortunately it is very hard work. A few
considerations:

1. Most demuxers are written in pull style and expect blocking AVIO
contexts. To use them in push style would require either rewriting them
quite in depth or running them in a separate context. There are too many
of them to make the first option practical. That leaves running in a
separate context, so that when they call avio_read() to get data, expect
blocking until all requested data is available, and keep state on the
stack, they actually yield the execution to the rest of the application.
Yet, the most common demuxers should probably be adapted to a push
design for robustness and efficiency.

2. For contexts, we have threads, but threads are somewhat expensive. A
portable implementation of contexts / coroutines in lavu would be
welcome IMHO.

3. All this applies not only to demuxers but to protocols too. This is
actually more difficult because some protocols need to open several
sub-protocols: you would need some way of registering several sockets
from the protocols to the event loop.

4. Some demuxers have sub-demuxers too, the API must be able to deal
with them.

5. Some protocols need to be run in a separate thread to ensure very low
latency (think UDP in high bitrate: the kernel could discard packets).
OTOH, the same thread can be used to run several low-latency protocols
if the overhead is low enough.

If you intend to work on that, I would be happy to help to the extent of
my ability and availability.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180320/1bdc356c/attachment.sig>