[FFmpeg-devel] [PATCH 2/4] avformat/tls_openssl: fix dtls_handshake return code

[Jack Lau]

> On Jul 9, 2025, at 22:14, Timo Rothenpieler <timo at rothenpieler.org> wrote:
> 
> On 09/07/2025 15:36, Jack Lau wrote:
>> If the handshake is still in progress, dtls_handshake should
>> return a positive status code.
> 
> Shouldn't dtls_open/start also be calling it in a loop then?

> I don't think it's expected that you might be needed to call the handshake function in a loop after a urlcontext was successfully opened.
It’s a special situation in WHIP.  The ICE, DTLS, SRTP reuse the same udp.
But the udp socket can’t be passed by FFmpeg option to DTLS, 
so I create a function(named ff_tls_set_external_socket now) and call it after dtls_open in WHIP implementation, 
dtls can’t handshake because the udp haven’t set
> 
> What I've done for the schannel implementation is force nonblocking off for the handshake, since there is just no good way to perform it in a nonblocking way, and you just always end up looping until it's done anyway.
But the handshake might work well using BLOCK mode then the dtls handshake will be finished in once function called(openssl will loop internal in BLOCK mode).

I’ll try it later.
> 
>> Signed-off-by: Jack Lau <jacklau1222 at qq.com>
>> ---
>>  libavformat/tls_openssl.c | 7 +++----
>>  1 file changed, 3 insertions(+), 4 deletions(-)
>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>> index 8639ac9758..ffd9cd51d2 100644
>> --- a/libavformat/tls_openssl.c
>> +++ b/libavformat/tls_openssl.c
>> @@ -716,15 +716,14 @@ static int openssl_dtls_verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
>>    static int dtls_handshake(URLContext *h)
>>  {
>> -    int ret = 0, r0, r1;
>> +    int ret = EINPROGRESS, r0, r1;
>>      TLSContext *p = h->priv_data;
>>        r0 = SSL_do_handshake(p->ssl);
>>      r1 = SSL_get_error(p->ssl, r0);
>>      if (r0 <= 0) {
>>          if (r1 != SSL_ERROR_WANT_READ && r1 != SSL_ERROR_WANT_WRITE && r1 != SSL_ERROR_ZERO_RETURN) {
>> -            av_log(p, AV_LOG_ERROR, "TLS: Read failed, r0=%d, r1=%d %s\n", r0, r1, openssl_get_error(p));
>> -            ret = AVERROR(EIO);
>> +            ret = print_ssl_error(h, r1);
>>              goto end;
>>          }
>>      } else {
>> @@ -734,7 +733,7 @@ static int dtls_handshake(URLContext *h)
>>      /* Check whether the DTLS is completed. */
>>      if (SSL_is_init_finished(p->ssl) != 1)
>>          goto end;
>> -
>> +    ret = 0;
>>      p->tls_shared.state = DTLS_STATE_FINISHED;
>>  end:
>>      return ret;
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".