[FFmpeg-devel] [PATCH] avformat/hls: Added subtitle support

Franklin Phillips franklinphillips at gmx.com
Wed Nov 16 15:25:00 EET 2016

On Wed, Nov 16, 2016 at 12:33:32PM +0100, wm4 wrote:
> On Wed, 16 Nov 2016 11:21:59 +0000
> Franklin Phillips <franklinphillips at gmx.com> wrote:
> > On Wed, Nov 16, 2016 at 11:41:41AM +0100, wm4 wrote:
> > > On Wed, 16 Nov 2016 10:31:10 +0000
> > > Franklin Phillips <franklinphillips at gmx.com> wrote:
> > >   
> > > > This patch is a fix for ticket #2833.
> > > > 
> > > > Each subtitle segment is its own WebVTT file and has to demuxed individually.  
> > >   
> > > > The timing of the subtitles are not perfect but it is the best I could  
> > > 
> > > What does this mean?
> > >   
> > When I convert a mp4 file with subtitles into hls and then back into mp4, the subtitles are off by a couple 100 milliseconds usually. It seems to be caused by the first audio/video packet not having a 0 timestamp causing ffmpeg to subtract the timestamp of that first packet from all the subsequent packets.
> FFmpeg determines a global start_time, which is probably used in
> transcoding to offset all streams or so. But that shouldn't lead to
> timestamp mismatches because it's the same offset.
> Can you check the timestamps with ffprobe? Also make sure you return
> subtitle packets ideally before audio/video packets with the same or a
> higher timestamp. Otherwise it could happen that a subtitle packet is
> too late to be rendered on video. (Though I don't know what happens
> when remuxing subtitles in this scenario.)
I think you are implying the subtitles are delayed, but that is not the case, in most cases I've found them to be early.

I just double checked what was happening and it is as follows:

demuxer -> ist_index:0 type:subtitle next_dts:NOPTS next_dts_time:NOPTS next_pts:NOPTS next_pts_time:NOPTS pkt_pts:127 pkt_pts_time:0.127 pkt_dts:127 pkt_dts_time:0.127 off:-1401000 off_time:-1.401
demuxer+ffmpeg -> ist_index:0 type:subtitle pkt_pts:-1274 pkt_pts_time:-1.274 pkt_dts:-1274 pkt_dts_time:-1.274 off:-1401000 off_time:-1.401
muxer <- type:subtitle pkt_pts:-1274 pkt_pts_time:-1.274 pkt_dts:-1274 pkt_dts_time:-1.274 size:45
demuxer -> ist_index:2 type:audio next_dts:NOPTS next_dts_time:NOPTS next_pts:NOPTS next_pts_time:NOPTS pkt_pts:126090 pkt_pts_time:1.401 pkt_dts:126090 pkt_dts_time:1.401 off:-1401000 off_time:-1.401
demuxer+ffmpeg -> ist_index:2 type:audio pkt_pts:0 pkt_pts_time:0 pkt_dts:0 pkt_dts_time:0 off:-1401000 off_time:-1.401
muxer <- type:audio pkt_pts:0 pkt_pts_time:0 pkt_dts:0 pkt_dts_time:0 size:407

This is showing the first two packets of a hls playlist, first one is the subtitle packet which has a timestamp of 0.127 (this is the timestamp that appears in the .vtt segment file so the demuxer appears to be working correctly), it is being offset by -1.401 which is the timestamp of the first audio packet. The problem is that the subtitle should appear 0.127 seconds from the start of the video and the offset is not needed. Now that I think about it, the issue may be with the hls muxer which was used to create these hls segments because the timestamps of the subtitles do not have the same point of reference as the audio/video packets.

> > > > do and it also does not take into account the X-TIMESTAMP-MAP header
> > > > in the WebVTT files which is needed to conform to the specification
> > > > (https://tools.ietf.org/html/draft-pantos-http-live-streaming-20#section-3.5).  
> > > 
> > > I'm not entirely familiar how HLS works. Shouldn't this be fine as long
> > > as the webvtt events have the correct "final" timestamp?
> > >   
> > Yes, I believe so, the fall back in the specification is to assume the value of that header is 0 meaning no offset and all the examples I've seen have that value set to 0 anyway. I don't think it's a crucial feature that needs to be done ASAP.
> > 
> > > > 
> > > > [...]
> > > > +  
> > > 
> > > That seems to duplicate a lot of code from the normal read path...
> > >   
> > I don't think there is a lot of duplication, and the duplication that there is, I don't think can be factored out cleanly due to the different ways the two methods are called and the difference in behaviour of the two methods.
> > 
> Well, if you say so. Still hurts a bit.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

More information about the ffmpeg-devel mailing list