[rtmpdump] rtmpdump / librtmp rtmps support broken for TLS 1.3 endpoints

[Francesc Pinyol Margalef]

Hi,
I have the same problem. I'm trying to send a live video to Facebook
rtmps (https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/1554).
I have seen in https://man7.org/linux/man-pages/man3/gnutls_handshake.3.html
that some responses should not be considered as critical.
I have compiled librtmp with gnutls and I have added some debug logs.
The response from Facebook is a -28 (GNUTLS_E_AGAIN). This means
that the connection should be retried until we get a 0. Something like:

diff --git a/librtmp/rtmp.c b/librtmp/rtmp.c
index 0865689..1fea3cd 100644
--- a/librtmp/rtmp.c
+++ b/librtmp/rtmp.c
@@ -976,17 +976,22 @@ RTMP_TLS_Accept(RTMP *r, void *ctx)
int
RTMP_Connect1(RTMP *r, RTMPPacket *cp)
{
+  int ret;
+
  if (r->Link.protocol & RTMP_FEATURE_SSL)
    {
#if defined(CRYPTO) && !defined(NO_SSL)
      TLS_client(RTMP_TLS_ctx, r->m_sb.sb_ssl);
      TLS_setfd(r->m_sb.sb_ssl, r->m_sb.sb_socket);
-      if (TLS_connect(r->m_sb.sb_ssl) < 0)
-       {
-         RTMP_Log(RTMP_LOGERROR, "%s, TLS_Connect failed", __FUNCTION__);
-         RTMP_Close(r);
-         return FALSE;
-       }
+      do {
+          ret = TLS_connect(r->m_sb.sb_ssl);
+         RTMP_Log(RTMP_LOGERROR, "%s, TLS_Connect returned: %d",
__FUNCTION__, ret);
+          if (gnutls_error_is_fatal(ret)) {
+              RTMP_Log(RTMP_LOGERROR, "%s, TLS_Connect failed %d",
__FUNCTION__, ret);
+              RTMP_Close(r);
+              return FALSE;
+         }
+      } while (ret);
#else
      RTMP_Log(RTMP_LOGERROR, "%s, no SSL/TLS support", __FUNCTION__);
      RTMP_Close(r);

Now I managed to get a connection, but the handshake fails:

0:00:00.186899858  3000 0x55e39c5c74a0 ERROR                   rtmp
:0:: RTMP_Connect1, TLS_Connect returned: -28
0:00:00.186995893  3000 0x55e39c5c74a0 ERROR                   rtmp
:0:: RTMP_Connect1, TLS_Connect returned: -28
0:00:00.188763869  3000 0x55e39c5c74a0 ERROR                   rtmp
:0:: RTMP_Connect1, TLS_Connect returned: 0
0:00:00.234868462  3000 0x55e39c5c74a0 ERROR                   rtmp
:0:: RTMP_Connect1, handshake failed.

I'll keep investigating.

Francesc

-- 
Francesc Pinyol Margalef
http://www.francescpinyol.cat/