I did not do much testing here, since I don't use rtmpdump. Also see https://polarssl.org/kb/how-to/migrate-from-polarssl-1.2-to-polarssl-1.3 for reference which also explains why ssl_set_ciphersuites() has been removed completely.