[rtmpdump] rtmpdump / librtmp rtmps support broken for TLS 1.3 endpoints

[Phil Cluff]

Hey there,

I've been down a rabbit hole the last couple of days debugging some broken
rtmp tools, specifically, gstreamer's rtmp plugin on Ubuntu 20.04.

I've narrowed this issue down to being librtmp not being able to talk to
servers that support TLS 1.3, when compiled against gnutls (openssl may
also be broken, but that doesn't seem to compile right now on master).

I've tested against 2 known good TLS 1.3 supporting servers, Mux's and
Facebook's, and in both cases the rtmp handshake fails. The handshake fails
in at this line in handshake.h:

if (!WriteN(r, (char *)clientsig-1, RTMP_SIG_SIZE + 1))
    return FALSE;

Changing the allowable TLS versions in the gnutls client fixes this
failure, allowing the same configuration to connect over TLS 1.2. For
example, by setting the following priorities in rtmp_sys.h:

#define TLS_client(ctx,s) gnutls_init((gnutls_session_t *)(&s),
GNUTLS_CLIENT); gnutls_priority_set_direct(s, "NORMAL:-VERS-TLS1.3", NULL);
gnutls_credentials_set(s, GNUTLS_CRD_CERTIFICATE, ctx->cred)

I've tested other rtmp implementations, and they're unaffected,
specifically rtmp2 in gstreamer, and the implementation in ffmpeg both work
fine with TLS 1.3.

This is unfortunate as it makes librtmp, rtmpdump, and any associated tools
unusable with rtmps servers on Ubuntu 20.04.

If anyone has any hints or pointers on why this might be failing, I'd love
to hear them. Happy to provide servers or endpoints for testing!

Thanks!

-- 
Phil Cluff | Streaming Architect | +44 7983 406 937 | phil at mux.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mplayerhq.hu/pipermail/rtmpdump/attachments/20200626/9816388d/attachment.html>