[FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Jack Lau]

> On Jun 5, 2025, at 15:02, Martin Storsjö <martin at martin.st> wrote:
> 
> On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:
> 
>> fix the missing data structure pkey in the tls_context
>> 
>> Signed-off-by: Jack Lau <jacklau1222 at qq.com>
>> ---
>> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
>> 1 file changed, 17 insertions(+), 13 deletions(-)
> 
> Thanks, this does fix the build break. However, I don't quite understand the fix...
> 
>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>> index b589d5d90a..bddeee9af8 100644
>> --- a/libavformat/tls_openssl.c
>> +++ b/libavformat/tls_openssl.c
>> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>>    TLSShared tls_shared;
>>    SSL_CTX *ctx;
>>    SSL *ssl;
>> +    EVP_PKEY *pkey;
>> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>>    BIO_METHOD* url_bio_method;
>> #endif
> 
> As far as I can see, nothing ever sets this new field, it is only used in a couple of places?
Thanks for the review.

The previous build error occurred because I forgot to properly set the EC_KEY when using OpenSSL versions earlier than 3.0.

In the current WHIP implementation, I initialize the key and certificate (either by reading from file or generating them) before the DTLS handshake, since the SDP requires fingerprints. The WHIP layer then passes the key and certificate content as strings into the DTLS context.

This fix ensures that the EVP_PKEY is loaded into the tls_context when DTLS starts. For OpenSSL versions below 1.0.2, we need to call SSL_CTX_set_tmp_ecdh, which requires an EC_KEY. So, i extract the EC_KEY from the EVP_PKEY.

I hope that explanation was clear—please feel free to reach out if you have any further questions.

> @@ -876,6 +877,9 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     int ret = 0;
>     c->is_dtls = 1;
>     const char* ciphers = "ALL";
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> +    EC_KEY *ec_key;
> +#endif
>     /**
>      * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c.
>      * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c.
> @@ -908,15 +912,6 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     }
> #endif
> 
> -#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> -#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> -    if (ctx->dtls_eckey)
> -        SSL_CTX_set_tmp_ecdh(p->ctx, p->dtls_eckey);
> -#else
> -    SSL_CTX_set_ecdh_auto(p->ctx, 1);
> -#endif
> -#endif
> -
>     /**
>      * We activate "ALL" cipher suites to align with the peer's capabilities,
>      * ensuring maximum compatibility.
> @@ -930,6 +925,17 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     ret = openssl_init_ca_key_cert(h);
>     if (ret < 0) goto fail;
> 
> +#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> +    if (p->pkey)
> +        ec_key = EVP_PKEY_get1_EC_KEY(p->pkey);
> +    if (ec_key)
> +        SSL_CTX_set_tmp_ecdh(p->ctx, ec_key);
> +#else
> +    SSL_CTX_set_ecdh_auto(p->ctx, 1);
> +#endif
> +#endif
> +
>     /* Server will send Certificate Request. */
>     SSL_CTX_set_verify(p->ctx, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, openssl_dtls_verify_callback);
>     /* The depth count is "level 0:peer certificate", "level 1: CA certificate",
> @@ -1015,9 +1021,7 @@ static av_cold int dtls_close(URLContext *h)
>     av_freep(&ctx->tls_shared.fingerprint);
>     av_freep(&ctx->tls_shared.cert_buf);
>     av_freep(&ctx->tls_shared.key_buf);
> -#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> -    EC_KEY_free(ctx->dtls_eckey);
> -#endif
> +    EVP_PKEY_free(ctx->pkey);
>     return 0;
> }
> 
> -- 
> 2.49.0

> 
> // Martin
Thanks
Jack
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".