[FFmpeg-user] On the recovery behavior of HLS muxer when slaved to tee'd fifo muxers in ffmpeg.

[Dennis Mungai]

On Wed, 25 Mar 2020 at 08:32, Dennis Mungai <dmngaie at gmail.com> wrote:

> On Wed, 25 Mar 2020 at 08:27, Dennis Mungai <dmngaie at gmail.com> wrote:
>
>> Four months ago, I posted a ticket on trac, #8387, see
>> https://trac.ffmpeg.org/ticket/8387
>>
>> Which described an anomaly with the recovery behavior of the underlying
>> HLS muxers called up via the tee's fifo slaves. More details in the ticket.
>>
>> What I've discovered by extensively studying this behavior is that when
>> dealing with fragmented mp4, the HLS muxer (as configured above) will
>> definitely fail to re-create the init.mp4 file should the target path (on a
>> local or remote filesystem) go missing.
>>
>> This is still present with ffmpeg git head, compiled a few hours ago.
>>
>> Case in point: Simulating failures with unstable NFS mounts, using the
>> same recovery options for fifo work flawlessly for DASH where the init
>> segments per variant are successfully re-created but the same doesn't carry
>> over for HLS.
>>
>> What gives?
>>
>> Are there specific tee and fifo muxer settings I can try out to resolve
>> this?
>>
>> Thanks,
>>
>> Dennis.
>>
>
> Update: A bit too soon, perhaps I should also update the subject.
> DASH demonstrates something similar:
> In effect, the muxer tries to re-create the init segment(s) for each
> variant stream(s), but fails with the ominous message:
>
> "Application provided invalid, non monotonically increasing dts to muxer
> in stream 0"
>
>

Update: I fixed the issue.

For NFS mounts, use these options in fstab:

$server_ip:/$nfs_mount /$mount_point nfs
timeo=21,fg,nofail,retrans=600,actimeo=3,vers=4,soft,x-systemd.automount,x-systemd.mount-timeout=infinity,retry=10000_netdev
0 0

Sneaky details are in the systemd NFS guide:
https://www.freedesktop.org/software/systemd/man/systemd.mount.html

And for FFmpeg, configure the muxer options as shown:

(a). For fifo muxers called up by tee, pass these options:
use_fifo=1:fifo_options=queue_size=90000\\:attempt_recovery=1\\:drop_pkts_on_overflow=1
(b). For both HLS and DASH, pass the option ignore_io_errors=1

This was tested with ffmpeg 1:N transcode to multiple NFS mounted
directories for each output, and can confirm clean recovery when NFS
server(s) are restarted.
I cannot (yet) explain why this setup works differently from a similar case
but with local filesystems, but as tested over the last two days, this is
what I got to work with no issues.

Leaving this here incase someone else runs into the same issue with NFS and
transparent tee/fifo recovery with ffmpeg and HLS output.