[FFmpeg-devel] [PATCH v3 0/9] Fix some issues in tls_openssl

[Jack Lau]

This patchset aims to fix some issues so that DTLS can work properly (not just for WHIP)

Quick experience using such command:
ffmpeg -v debug -listen 1 -cert_file cert.pem -key_file key.pem -mtu 1500 -i dtls://127.0.0.1:4444 -c copy -y out.mp4
ffmpeg -v trace -f lavfi -re -i testsrc2=duration=5:size=128x72:rate=30 -an -sn -c:v libx264 -mtu 10000 -f mpegts "dtls://127.0.0.1:4444"

Or using avio public api:
https://github.com/JackLau1222/openssl-dtls-bio-example/tree/master/ffmpeg_case

I notice DTLS still have to do some cleanups, I plan to implement WHIP DTLS client role next step so i'll do it also in coming days

Original description:
This patchset aims to fix some issues when i try to utilize DTLS using avio.
I create a simple DTLS client and server case here
https://github.com/JackLau1222/openssl-dtls-bio-example/tree/master/ffmpeg_case

This patchset fix:
1. dtls_handshake can't return positive code when it still in progressing
2. udp server mode haven't dest_addr so we need set it through last_recv_addr
3. some code cleanup

This patchset depends on Timo's latest schannel patchset
More details: https://github.com/BtbN/FFmpeg/pull/3

Jack Lau (9):
  avformat/tls: add trace function for log TLS/DTLS record
  avformat/tls_openssl: fix dtls_handshake return code
  avformat/tls_openssl: remove all redundant "TLS: " in log with AVClass
  avformat/tls_openssl: make tls and dtls use one close function
  avformat/whip: free udp socket after dtls free
  avformat/tls_openssl: replace 1 to TLS_ST_OK to be more clear
  avformat/tls_openssl: remove requirement for dtls must init cert and
    key
  avformat/tls_openssl: auto set the dest addr when dtls in listen mode
  avformat/tls_openssl: init DTLS context with explicit method

 libavformat/tls.c         | 56 +++++++++++++++++++++++++++
 libavformat/tls.h         |  2 +
 libavformat/tls_openssl.c | 81 ++++++++++++++++++++-------------------
 libavformat/whip.c        |  2 +-
 4 files changed, 100 insertions(+), 41 deletions(-)

-- 
2.49.0