[FFmpeg-user] http_persistent not honored when encryption is enabled

[Valentin Schweitzer]

Hi,

Enabling http_persistent and encryption in the hls muxer leads to HTTP
connections that multiple Web servers cannot deal with properly.
In particular, ffmpeg sends HTTP PUT requests with a "Connection: keep-alive"
header for each segment file, but it closes the connection right
afterwards and opens a new one for the next segment. I can observe this
behavior with Wireshark and multiple different Web servers, e.g., Apache,
Kestrel and tornado.
Using the most recent zeranoe build of ffmpeg for 64-bit Windows,

.\ffmpeg -y -hide_banner -f lavfi -i color=c=black:s=1920x1080:r=25 -vcodec libx264 -f hls -hls_segment_type mpegts -hls_time 0.04 -hls_list_size 100 -hls_flags second_level_segment_index+split_by_time+program_date_time -hls_init_time 0.04 -http_persistent 1 -strftime 1 -hls_segment_filename http://127.0.0.1:45000/%%18d.ts -hls_enc 1 -hls_enc_key fedcba9876543210 -hls_enc_iv fedcba9876543210 -method PUT http://127.0.0.1:45000/index.m3u8

shows the described behavior of connections being closed despite
http_persistent being enabled and keep-alive being sent. With loglevel
debug, it is visible that a connection is started for each segment,
but the closing of the connection is only implicitly visible due to
the statistics being printed (the closing can be seen more clearly
with the FIN ACK in Wireshark):

[..]
[hls @ 000001cf588da0c0] Opening 'crypto:http://127.0.0.1:45000/streams/-/data/1080p/000000000000000011.ts' for writing
[crypto @ 000001cf588e6f80] No default whitelist set
[http @ 000001cf588e6180] Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
[tcp @ 000001cf588e7540] Original list of addresses:
[tcp @ 000001cf588e7540] Address 127.0.0.1 port 45000
[tcp @ 000001cf588e7540] Interleaved list of addresses:
[tcp @ 000001cf588e7540] Address 127.0.0.1 port 45000
[tcp @ 000001cf588e7540] Starting connection attempt to 127.0.0.1 port 45000
[tcp @ 000001cf588e7540] Successfully connected to 127.0.0.1 port 45000
[http @ 000001cf588e6180] request: PUT /streams/-/data/1080p/000000000000000011.ts HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.35.100
Accept: */*
Connection: keep-alive
Host: 127.0.0.1:45000
Icy-MetaData: 1

[AVIOContext @ 000001cf59a466c0] Statistics: 0 seeks, 1 writeouts
[hls @ 000001cf588da0c0] Opening 'http://127.0.0.1:45000/streams/-/data/1080p/index.m3u8' for writing
[..]

When encryption is disabled,

.\ffmpeg -y -hide_banner -f lavfi -i color=c=black:s=1920x1080:r=25 -vcodec libx264 -f hls -hls_segment_type mpegts -hls_time 0.04 -hls_list_size 100 -hls_flags second_level_segment_index+split_by_time+program_date_time -hls_init_time 0.04 -http_persistent 1 -strftime 1 -hls_segment_filename http://127.0.0.1:45000/%%18d.ts -method PUT http://127.0.0.1:45000/index.m3u8

the connection is left open as it should be. This is also
visible with loglevel debug (note the missing statistics
for the closed connection):

[..]
[http @ 000001f9bd0e7880] Opening 'http://127.0.0.1:45000/streams/-/data/1080p/000000000000000011.ts' for writing
[http @ 000001f9ba626d40] request: PUT /streams/-/data/1080p/000000000000000011.ts HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.35.100
Accept: */*
Connection: keep-alive
Host: 127.0.0.1:45000
Icy-MetaData: 1

[http @ 000001f9bd0e7880] Opening 'http://127.0.0.1:45000/streams/-/data/1080p/index.m3u8' for writing
[..]

*QUESTION*:
How can I get ffmpeg to keep the connection open also when encryption
is enabled?

Greetings,
Valentin

PS.: Looking at the source code, the connection closing logic in hlsenc_io_close
(https://github.com/FFmpeg/FFmpeg/blob/bb01baae269cd2d7a0e61dc9bf327168edd43b04/libavformat/hlsenc.c#L273)
seems to differ from the opening logic in hlsenc_io_open
(https://github.com/FFmpeg/FFmpeg/blob/bb01baae269cd2d7a0e61dc9bf327168edd43b04/libavformat/hlsenc.c#L251).
The difference seems to be related only to encryption. However, if I remove
"|| hls->key_info_file || hls->encrypt" in hlsenc_io_close to make the it symmetric
to hlsenc_io_open, the connections are still closed, but with slightly different
packets in the lower TCP/IP layers. The issue remains.