[FFmpeg-devel] Status and Plans for Subtitle Filters

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Nicolas George
> Sent: Monday, February 17, 2020 8:37 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] Status and Plans for Subtitle Filters
> 
> Soft Works (12020-02-14):
> > I am looking for some guidance regarding future plans about processing
> > subtitle streams in filter graphs.
> >
> > Please correct me where I'm wrong - this is the situation as I've
> > understood it so far:
> >
> > - Currently, ffmpeg filter graphs do not support processing subtitle
> > streams
> > - This is why filters like 'subtitles' and 'ass' need to open, read
> > and parse the media file a second time  in parallel instead of just
> > taking the subtitle events from ffmpeg'demuxing
> > - For graphical subtitles, there exists the so-called 'sub2video'
> > workaround which is injecting the the graphical subtitle overlay
> > images into the video filtergraph by declaring them as video
> > - The 'sub2video' was meant to exist until filtering would support
> > subtitle streams
> > - A while ago, Clement Boesch submitted a patch for adding subtitle
> > streams to filtergraph processing
> >
> (https://patchwork.ffmpeg.org/project/ffmpeg/patch/20161102220934.2601
> > 0-1-u at pkh.me/)
> > - I read through all the discussion about it, but nothing happened
> > afterwards and I couldn't find any indication about why it didn't get
> > merged
> 
> This looks accurate.
> 
> > I'm asking because I'm intending to implement a subtitle filter that
> > operates on in-stream data rather than a separate input and that will
> > render text on transparent frames for later overlay.
> >
> > The possible options that I have identified so far for creating that
> > kind of "subtitle rendering filter" would be:
> >
> > - Create a video source filter and implement some hack to get the
> > subtitle data from the decoder to that filter - or...
> > - Jump on the sub2video implementation and extend it to render overlay
> > images  in case of text subtitles - or...
> > - check out the situation with regards to adding subtitle filter
> > support in ffmpeg and ask about plans for this
> >
> > That's where I stand right now. Does  it even make sense, the way I
> > summarized it?`
> 
> I don't speak for the project as a whole, but I am quite confident that "some
> hack" would be accepted if and only if it is generic enough to be useful for
> many people, not just your use case. And tweaks to sub2video are "some
> hack" too.
> 
> If you want to implement real support for subtitles, that would be greatly
> appreciated, but I have to warn you it is a very difficult and intensive task.
> Otherwise it would have already been done. I can summarize where the
> difficulty resides:
> 
> - lavfi works with AVFrame, therefore subtitles need to be encoded into
>   AVFrame. This is the work of Clément that you found; he only started
>   on the rest.
> 
> - lavfi is not ready to have a third media type: there are parts that
>   strongly assume audio or video, and parts that merge the audio and
>   video case but cannot handle a different type.
> 
> - The utility filters that only work on metadata, like setpts, need to
>   be ported ported to the new media type. We don't want too much code
>   duplication, a more elegant approach needs to be found. Possibly make
>   the media type part of a first round of format negotiation.
> 
> - We need to decide the format negotiation for filters. Do we
>   automatically insert text→bitmap renderer or does it need to be
>   explicit?
> 
> - Subtitles streams are sparse, lavfi is not designed for that, and it
>   is a problem when subtitles are interleaved with audio or video. The
>   best way to solve it is dummy heartbeat frames. They can be
>   automatically generated by the framework with an API to connect two
>   buffersrc: when a video frame is added on one, generate a heartbeat
>   subtitle frame on the other if necessary.
> 
> Feel free to ask details on any of these points: I do not have the courage to
> start working on it for now, but I have thought about it.

Hi Nicholas,

Thanks a lot for your comments. To be honest, I pretty much feel the same
about it. It was painful enough to read through the discussion about Clement's
patch, and I'm neither ready for this nor do I have the time.

My idea would be rather to take at look at the latest work from Clement and
see whether it is in a state that is close to what I need - from your lines it
rather seems that it is not.

I guess my best bet will be to jump on the sub2video mechanism and extend
it to work with text subtitles. Each time the text changes, I create a new 
transparent image using the code from the subtitles filter. Then I let
sub2video process that image in the same way it does for graphical subs
images.

Does that sound reasonable to you?

PS: Obviously, the most appealing part of this is that it would involve a rather
limited amount of work..