[FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Jack Lau]

> On Jun 5, 2025, at 19:20, Martin Storsjö <martin at martin.st> wrote:
> 
> On Thu, 5 Jun 2025, Jack Lau wrote:
> 
>>> On Jun 5, 2025, at 15:02, Martin Storsjö <martin at martin.st> wrote:
>>> On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:
>>>> fix the missing data structure pkey in the tls_context
>>>> Signed-off-by: Jack Lau <jacklau1222 at qq.com>
>>>> ---
>>>> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
>>>> 1 file changed, 17 insertions(+), 13 deletions(-)
>>> Thanks, this does fix the build break. However, I don't quite understand the fix...
>>>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>>>> index b589d5d90a..bddeee9af8 100644
>>>> --- a/libavformat/tls_openssl.c
>>>> +++ b/libavformat/tls_openssl.c
>>>> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>>>>   TLSShared tls_shared;
>>>>   SSL_CTX *ctx;
>>>>   SSL *ssl;
>>>> +    EVP_PKEY *pkey;
>>>> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>>>>   BIO_METHOD* url_bio_method;
>>>> #endif
>>> As far as I can see, nothing ever sets this new field, it is only used in a couple of places?
>> Thanks for the review.
>> 
>> The previous build error occurred because I forgot to properly set the EC_KEY when using OpenSSL versions earlier than 3.0.
>> 
>> In the current WHIP implementation, I initialize the key and certificate (either by reading from file or generating them) before the DTLS handshake, since the SDP requires fingerprints. The WHIP layer then passes the key and certificate content as strings into the DTLS context.
>> 
>> This fix ensures that the EVP_PKEY is loaded into the tls_context when DTLS starts. For OpenSSL versions below 1.0.2, we need to call SSL_CTX_set_tmp_ecdh, which requires an EC_KEY. So, i extract the EC_KEY from the EVP_PKEY.
>> 
>> I hope that explanation was clear—please feel free to reach out if you have any further questions.
> 
> No that didn't answer my question.
> 
> As far as I can see, nothing sets the context variable p->pkey. It is used in openssl_init_ca_key_cert and later in dtls_start. But nothing ever sets p->key, so it will be NULL everywhere.
> 
> Did you test this code with openssl 1.0.2 (which those codepaths are for)?
> 
> It looks to me like this maybe should have an assignment in openssl_init_ca_key_cert, setting "p->key = pkey;" maybe?
Thanks for your reminder! 
I’ve sent the patch v2 that fix this issue.

And I tested the major openssl versions (1.0.1, 1.0.2, 1.1.0, 3.0, latest) and it works well.

> 
> // Martin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe”.
Thanks
Jack