[FFmpeg-user] Unable to record VP8+Opus from RTP+SDP

[Ted Park]

> If you notice, there is a lot of metadata about the RTP session (ssrc,
> cname), some about the ICE protocol (for WebRTC), but there isn't much
> information there about the specifics of the codecs.

I think that statement is rather specious… WebRTC relies on some other app (perhaps a browser webapp implemented in js) to negotiate those specifics.

> In this case, the
> preferred audio payload type is 111, which corresponds to the OPUS
> codec. The specification "opus/48000/2" is expected to already convey
> enough information to let the other peer know what codec it is going to
> receive, and it allows for some (much) variance in the actual encoding
> parameters that are going to be used; the receiver is just able to
> auto-detect those and decode the stream.

That’s just “opus/48000/2” being the only string you’re allowed to put for the subtype/clock/channel fields of rtpmap. The actual encoding might differ, but as you said, in most cases the receiver is able to use the in-band signaling to decode. But in this case you’re trying to codec copy using parameters ffmpeg determined solely from the SDP file.

>   v=0
>   o=- 6323209318560568134 2 IN IP4 127.0.0.1
>   s=-
>   t=0 0
>   a=group:BUNDLE 0 1
>   a=msid-semantic: WMS XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
...
>   a=mid:0
>   a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
>   a=extmap:2
>   http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
>   a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
>   a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
>   a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
>   a=sendrecv
>   a=msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   816c42db-7d97-4053-86a0-bae0e5010985
>   a=rtcp-mux
>   a=rtpmap:111 opus/48000/2
>   a=rtcp-fb:111 transport-cc
>   a=fmtp:111 minptime=10;useinbandfec=1
...
>   a=ssrc:3169356447 cname:XZAw22pKY7Y9bCph
>   a=ssrc:3169356447 msid:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   816c42db-7d97-4053-86a0-bae0e5010985
>   a=ssrc:3169356447 mslabel:XAb2pgeSAM6YZw6uCRj11AE5Raf3cKHPNGl0
>   a=ssrc:3169356447 label:816c42db-7d97-4053-86a0-bae0e5010985


WebRTC is far more complex than simple RTP. A separate application signals metadata about the connection and whether streams/tracks change or are added.
The extmap attributes declare the mapping between the RTP header extensions and the corresponding parameters that will be used in the session, which may well include framing parameters (for VBR, DTX, etc.)

And the fmtp attribute actually does specify formatting info...

> For the cases where the codec has really incompatible profiles, to the
> point where trying to decode with the wrong profile would fail, then
> that information is also transmitted in the SDP; this is the case of the
> H.264 profile levels (Baseline, Main, High, etc.), which are then
> accordingly transmitted in the SDP Offer as totally different Payload
> Types. But you'll notice that other codecs, like VP8, don't really need
> such distinction.

As with opus, the “extradata” that ffmpeg complains is missing would be provided from a separate signaler to WebRTC.

>   m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125
>   107 108 109 124 120 123
...
>   a=rtpmap:96 VP8/90000
>   a=rtcp-fb:96 goog-remb
>   a=rtcp-fb:96 transport-cc
>   a=rtcp-fb:96 ccm fir
>   a=rtcp-fb:96 nack
>   a=rtcp-fb:96 nack pli
>   a=rtpmap:97 rtx/90000
>   a=fmtp:97 apt=96

Also, as this session uses the AVPF profile, it can do things, like make sure the timestamp increments are synchronized with the RTP clock, get a keyframe at the start, have the server retransmit lost packets, etc.

> Of course, such an SDP Offer is not really missing any crucial detail;
> another browser such as Chrome, Firefox, Safari, etc. will be able to
> receive and decode the incoming audio and video. So what I'm really
> trying to do is doing the same with FFmpeg... now if you compare
> Chrome's SDP with mine, it turns out mine does already have all relevant
> details that are included in a "full-blown" SDP message.

You mentioned mediasoup. Can you have it serve the sdp or get the parameters and use those to write a proper one? I don’t know how much access you have to the media soup router, is it actually on localhost?