[FFmpeg-user] ffmpeg retry failed segments

Wed Sep 27 20:01:00 EEST 2023

I was actually able to reproduce this and see what it wrote in the logs.

This is the error it gets when reading first ts segment from the playlist:

[https @ 0000022951e4eec0] Stream ends prematurely at 2097152, should be
4743992s speed=N/A
[https @ 00000229522014c0] Opening '
https://heiod68-httpcache0-47115-cacheod0.dna.ip-only.net/47115-cacheod1/1808568/0/hls/fuha42000101/572917154-956364685-1800000.ts?version_hash=e18e4dc4'
for reading
[hls @ 0000022951d7d8c0] Opening '
https://heiod68-httpcache0-47115-cacheod0.dna.ip-only.net/47115-cacheod1/1808568/0/hls/fuha42000101/572917154-956364685-1800000.ts?version_hash=e18e4dc4'
for reading
[mpegts @ 0000022951e4eac0] Packet corrupt (stream = 0, dts = 369000).
[hls @ 0000022951d7d8c0] Packet corrupt (stream = 0, dts = 367200).
https://heiod68-httpcache0-47115-cacheod0.dna.ip-only.net/47115-cacheod1/1808568/0/hls/fuha42000101/572917154-956364685-prog_index.m3u8?version_hash=e18e4dc4:
corrupt input packet in stream 0

(I have anonymized the stream urls)

And then it continues to read the next file in the m3u8. But this causes a
freeze in the start of my video. Luckily it aborts if I use the -xerror
parameter, but I would like it to retry reading the actual stream instead.
The m3u8 file contains "thousands of files", so I cannot manually "warm it
up either". I have a utility that I use to download some streams, so manual
work is not practical.

On Wed, Sep 27, 2023 at 2:26 PM Torsten Kaiser <info at linux-rebellen.de>
wrote:

> Since the freeze seems to appear only with the first entry in the
> playlist, within the first download, you also could try to "plow the
> road" before actually trying to download via ffmpeg.
>
> e.g. something like this:
>
> wget --spider <m3u8-URL>
>
> sleep 1
>
> ffmpeg <your parameters>
>
> In spider-mode wget doesn't actually download the playlist, but tries to
> reach it prior to your invocation of ffmpeg. This way the route across
> the web is already scouted before ffmpeg tries to reach for the first
> input file in the list
>
> Honestly, I don't really expect ffmpeg to have a parameter to wait for
> the outside world, since the "ff" in ffmpeg means "fast forward"
>
> Good luck!
>
>
> On 27.09.23 13:48, Stein Rune Risa wrote:
> > Ok - I think I will investigate the output from ffmpeg for some of the
> > failed cases to see what happens. But ffmpeg does somehow skip some
> > segments, but next time they might be OK.
> >
> > I also learnt that I can use xerror parameter to stop if it encounters
> > errors. Then it is easy to detect it without visually inspect it.
> >
> >
> >
> > On Wed, Sep 27, 2023 at 1:44 PM Peter van den Houten <petervdh at gmail.com
> >
> > wrote:
> >
> >> Hi, reply below.
> >>
> >> On 27/09/2023 11:03, Torsten Kaiser wrote:
> >>> Waiting longer might not be helping. What if the first entry is indeed
> >> up to 5 seconds long, then immediately switching the source to the next
> >> URL? New URL, new TLS-handshake. Browsers do indeed wait longer and
> retry
> >> multiple times to get a lock to the next bit of any web content.
> Visible to
> >> the user by the "hour glass".
> >>> But: Knowing the location of the source m3u8 could provide the
> >> opportunity to download the playlist itself via [wget <m3u8-url>]. The
> >> following link to Wikipedia describes the structure of m3u-files.
> >>> https://en.wikipedia.org/wiki/M3U
> >>>
> >>> So it's all plain text. Media entries are local relative links or URLs,
> >> so you may want to keep the m3u8's location
> >>> Replacing the m3u8-filename with one of the media entries in question
> >> (if local append whole path, otherwise replace the whole construct with
> the
> >> URL-entry) should be a valid URL to the media source, provided the
> issuer
> >> allows access without evaluating the http-referer as being of its own
> realm.
> >>> So prepared, try to use ffmpeg with this single media file as input url
> >> , watching for errors.
> >>> No solution, but trying to circumvent suspected problems with certain
> >> entries in the playlist, narrowing down the problem
> >>> (And learned for myself what hides behind HLS streaming)
> >>>
> >>> On 27.09.23 09:00, Stein Rune Risa wrote:
> >>>> Hi!
> >>>>
> >>>> This is a very stable and well known source, so the network is also
> >> stable.
> >>>> There are never any issues streaming through their official player.
> >>>>
> >>>> I also see that the streams are delivered through well known CDN - so
> it
> >>>> should be fine.
> >>>>
> >>>> But I will try the xerror option. It is better to detect errors than
> >>>> getting some problem with the file.
> >>>>
> >>>> It also seems that the hiccup always happens within the first 5
> seconds
> >> of
> >>>> the stream - so there might be some "initial delay" when initializing
> >> the
> >>>> stream. Is there some parameter for ffmpeg to kind of allow longer
> >> response
> >>>> times from the source?
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Sep 27, 2023 at 8:16 AM Torsten Kaiser <
> info at linux-rebellen.de>
> >>>> wrote:
> >>>>
> >>>>> Looks like you download a known playlist from some unstable network
> >>>>> connection.
> >>>>>
> >>>>> To get a grasp on this: If you set the -xerror switch like
> >>>>>
> >>>>> ffmpeg -y -xerror -i https://file.com/myfiles.m3u8 -c copy
> >> /tmp/test.mkv
> >>>>> what happens? Download should stop at the first buggy/missing segment
> >> of
> >>>>> the playlist. Now repeat the download multiple times to see if the
> >>>>> problem is random or not.
> >>>>>
> >>>>> Missing segments could be a DRM problem or invalid/outdated entries
> in
> >>>>> the playlist. If the download always misses the same segment it's
> most
> >>>>> likely the segment entry is invalid.
> >>>>>
> >>>>> Buggy segments are most likely either already buggy in the source or
> a
> >>>>> network bottleneck. If the faults always occur on the same spot, then
> >>>>> the source itself is most likely already buggy. Encountered this
> myself
> >>>>> on legacy sources digitized from VHS tapes. Sometimes content is more
> >>>>> important than quality.
> >>>>>
> >>>>>
> >>>>> On 26.09.23 22:36, Stein Rune Risa wrote:
> >>>>>> I am using ffmpeg to download some streaming to a mkv file. The
> >> stream parts comes in a m3u8 file.
> >>>>>> ffmpeg -y -i https://file.com/myfiles.m3u8 -c copy /tmp/test.mkv
> >>>>>>
> >>>>>> In some cases, there have been some issues downloading some of the
> >> segments, and the mkv files have parts where it freezes for some seconds
> >> before continuing. I am now forced to check the file manually (by
> watching)
> >> and then redownload if it has any issues.
> >>>>>> I guess this happens because ffmpeg temporarily is not able to
> >> download some of the segments referred to in the m3u8?
> >>>>>> Is there some way that I can force ffmpeg to retry failed segments?
> >> Or at least abort if it happens? I am kind of looking for a way to get
> rid
> >> of themanual checking of the content.
> >>
> >>
> >> As Torsten pointed out, the problem could be DRM or outdated/unavailable
> >> segments. I have had no problems using the following code, with a user
> >> agent string which keeps the website happy and downloads all segments
> from
> >> most websites, maybe it will help with your instance. This example is to
> >> download a copyright-free movie from PlexVideos:
> >>
> >> ffmpeg -user_agent "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X)
> >> AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d
> >> Safari/8536.25" -i "
> >>
> https://vod-content.plexvideos.com/streams/62be17835e5872e950b87398/0f59c0e3-59ad-4362-8cf2-11f8508c46f2/stream_6.m3u8
> "
> >> -c copy -y the_brain_that_wouldn't_die.mkv.
> >>
> >> _______________________________________________
> >> ffmpeg-user mailing list
> >> ffmpeg-user at ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
> >>
> > _______________________________________________
> > ffmpeg-user mailing list
> > ffmpeg-user at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>