[PATCH] Better URL decoding support
Currently RtmpDump has limited URL decoding support. If you use Bastien’s patch as shown here http://lists.mplayerhq.hu/pipermail/rtmpdump/2012-September/002093.html You can pass encoded characters like this rtmpdump -o a.flv -i "rtmp://server.com/app playpath=foobar\0a" Where "\0a" will be decoded as a newline. Also the Playpath can be decoded using an unpatched RtmpDump, as long as it is part of the "-r" parameter and properly parsed. Example rtmpdump -o a.flv -r rtmp://server.com/app/foobar%0a Where "%0a" will be decoded as a newline. This presents two problems. One, many times RtmpDump is unable to correctly parse the Playpath, mixing it with the "app". Two, if you explicitly state the Playpath with "-y", you lose the URL decoding support. I have simply moved the URL decoding function that Howard wrote to parseurl.c and declared it in rtmp.h, so that it can be used across the project. --- librtmp/parseurl.c | 73 +++++++++++++++++++++++++++++++++++++--------------- librtmp/rtmp.h | 1 + rtmpdump.c | 1 + rtmpgw.c | 47 --------------------------------- 4 files changed, 54 insertions(+), 68 deletions(-) diff --git a/librtmp/parseurl.c b/librtmp/parseurl.c index 57abfe7..07fbbb9 100644 --- a/librtmp/parseurl.c +++ b/librtmp/parseurl.c @@ -206,7 +206,7 @@ void RTMP_ParsePlaypath(AVal *in, AVal *out) { const char *playpath = in->av_val; const char *temp, *q, *ext = NULL; const char *ppstart = playpath; - char *streamname, *destptr, *p; + char *streamname, *destptr; int pplen = in->av_len; @@ -265,26 +265,57 @@ void RTMP_ParsePlaypath(AVal *in, AVal *out) { } } - for (p=(char *)ppstart; pplen >0;) { - /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - } - if (*p == '%') { - unsigned int c; - sscanf(p+1, "%02x", &c); - *destptr++ = c; - pplen -= 3; - p += 3; - } else { - *destptr++ = *p++; - pplen--; - } - } - *destptr = '\0'; + memcpy(destptr, ppstart, pplen * sizeof(char)); + destptr[pplen] = '\0'; + http_unescape(streamname); out->av_val = streamname; - out->av_len = destptr - streamname; + out->av_len = strlen(streamname); +} + +/* inplace http unescape. This is possible .. strlen(unescaped_string) <= strlen(esacped_string) */ +void +http_unescape(char *data) +{ + char hex[3]; + char *stp; + int src_x = 0; + int dst_x = 0; + + int length = (int) strlen(data); + hex[2] = 0; + + while (src_x < length) + { + if (strncmp(data + src_x, "%", 1) == 0 && src_x + 2 < length) + { + // + // Since we encountered a '%' we know this is an escaped character + // + hex[0] = data[src_x + 1]; + hex[1] = data[src_x + 2]; + data[dst_x] = (char) strtol(hex, &stp, 16); + dst_x += 1; + src_x += 3; + } + else if (src_x != dst_x) + { + // + // This doesn't need to be unescaped. If we didn't unescape anything previously + // there is no need to copy the string either + // + data[dst_x] = data[src_x]; + src_x += 1; + dst_x += 1; + } + else + { + // + // This doesn't need to be unescaped, however we need to copy the string + // + src_x += 1; + dst_x += 1; + } + } + data[dst_x] = '\0'; } diff --git a/librtmp/rtmp.h b/librtmp/rtmp.h index bd85a17..242a536 100644 --- a/librtmp/rtmp.h +++ b/librtmp/rtmp.h @@ -285,6 +285,7 @@ extern "C" unsigned int *port, AVal *playpath, AVal *app); void RTMP_ParsePlaypath(AVal *in, AVal *out); + void http_unescape(char *data); void RTMP_SetBufferMS(RTMP *r, int size); void RTMP_UpdateBufferMS(RTMP *r); diff --git a/rtmpdump.c b/rtmpdump.c index e52f7d4..a6cbdfd 100644 --- a/rtmpdump.c +++ b/rtmpdump.c @@ -963,6 +963,7 @@ main(int argc, char **argv) } break; case 'y': + http_unescape(optarg); STR2AVAL(playpath, optarg); break; case 'Y': diff --git a/rtmpgw.c b/rtmpgw.c index 0cf56bb..50b8c2a 100644 --- a/rtmpgw.c +++ b/rtmpgw.c @@ -220,53 +220,6 @@ FILE *netstackdump = NULL; FILE *netstackdump_read = NULL; #endif -/* inplace http unescape. This is possible .. strlen(unescaped_string) <= strlen(esacped_string) */ -void -http_unescape(char *data) -{ - char hex[3]; - char *stp; - int src_x = 0; - int dst_x = 0; - - int length = (int) strlen(data); - hex[2] = 0; - - while (src_x < length) - { - if (strncmp(data + src_x, "%", 1) == 0 && src_x + 2 < length) - { - // - // Since we encountered a '%' we know this is an escaped character - // - hex[0] = data[src_x + 1]; - hex[1] = data[src_x + 2]; - data[dst_x] = (char) strtol(hex, &stp, 16); - dst_x += 1; - src_x += 3; - } - else if (src_x != dst_x) - { - // - // This doesn't need to be unescaped. If we didn't unescape anything previously - // there is no need to copy the string either - // - data[dst_x] = data[src_x]; - src_x += 1; - dst_x += 1; - } - else - { - // - // This doesn't need to be unescaped, however we need to copy the string - // - src_x += 1; - dst_x += 1; - } - } - data[dst_x] = '\0'; -} - TFTYPE controlServerThread(void *unused) { -- 1.7.9
On Tue, Oct 23, 2012 at 9:12 AM, Steven Penny <svnpenn@gmail.com> wrote:
- /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - }
Please be careful not to break things. Anyways, the proper behavior is to actually *NOT* do any urldecoding. The %xx should be sent to the server as-is. This is what the Flash Player does and is the same as how normal HTTP clients handle URLs.
On Tue, Oct 23, 2012 at 11:39 PM, NhJm wrote:
- /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - }
Please be careful not to break things.
If you are certain this is an error, please state as such. In the mean time I will assume it is and check it.
Anyways, the proper behavior is to actually *NOT* do any urldecoding. The %xx should be sent to the server as-is.
This has not been my experience. Admittedly I have only been testing with that one server, but it is not accepting encoded characters. Notice carefully the literal newlines in the first example. WORKS $ rtmpdump -o a.flv -r rtmp://freeview.fms.visionip.tv/live/tvnetwork-hellenict v-sigma-hsslive-25f-4x3-SDh%0a -B 1 -V DEBUG: Hostname : freeview.fms.visionip.tv DEBUG: Port : 1935 DEBUG: Playpath : tvnetwork-hellenictv-sigma-hsslive-25f-4x3-SDh DEBUG: tcUrl : rtmp://freeview.fms.visionip.tv:1935/live DEBUG: app : live ... DEBUG: Property: <Name: details, STRING: tvnetwork-hellenictv-sig ma-hsslive-25f-4x3-SDh
DEBUG: (object end) DEBUG: (object end) DEBUG: HandleInvoke, server invoking <onStatus> DEBUG: HandleInvoke, onStatus: NetStream.Play.Stop DEBUG: Invoking deleteStream DEBUG: Got Play.Complete or Play.Stop from server. Assuming stream is complete 46.088 kB / 1.63 sec DEBUG: RTMP_Read returned: 0 Download complete DEBUG: Closing connection. NOT WORK $ rtmpdump -o a.flv -r rtmp://freeview.fms.visionip.tv/live -y tvnetwork-hellen ictv-sigma-hsslive-25f-4x3-SDh%0a -B 1 -V DEBUG: Hostname : freeview.fms.visionip.tv DEBUG: Port : 1935 DEBUG: Playpath : tvnetwork-hellenictv-sigma-hsslive-25f-4x3-SDh%0a DEBUG: tcUrl : rtmp://freeview.fms.visionip.tv:1935/live DEBUG: app : live ... DEBUG: Property: <Name: description, STRING: rtmp://rtmp-origin-01.fm s.visionip.tv:1935/rtmpinput1/_definst_/tvnetwork-hellenictv-sigma-hsslive-25f-4 x3-SDh%0a|rtmp://rtmp-origin-02.fms.visionip.tv:1935/rtmpinput1/_definst_/tvnetw ork-hellenictv-sigma-hsslive-25f-4x3-SDh%0a is now unpublished> DEBUG: Property: <Name: clientid, NUMBER: 860643237.00> DEBUG: (object end) DEBUG: (object end) DEBUG: HandleInvoke, server invoking <onStatus> DEBUG: HandleInvoke, onStatus: NetStream.Play.UnpublishNotify DEBUG: Invoking deleteStream DEBUG: Closing connection.
On Wed, Oct 24, 2012 at 12:10 AM, Steven Penny <svnpenn@gmail.com> wrote:
On Tue, Oct 23, 2012 at 11:39 PM, NhJm wrote:
- /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - }
Please be careful not to break things.
If you are certain this is an error, please state as such. In the mean time I will assume it is and check it.
rtmpdump's current behavior is to translate "blah.mp4?foo" into "mp4:blah?foo". By removing this piece of code, rtmpdump will instead generate "mp4:blah.mp4?foo"
Anyways, the proper behavior is to actually *NOT* do any urldecoding. The
%xx should be sent to the server as-is.
This has not been my experience. Admittedly I have only been testing with that one server, but it is not accepting encoded characters. Notice carefully the literal newlines in the first example.
The official Flash Player does not urldecode the path like rtmpdump is currently doing to part of the URL. Using http://support.akamai.com/flash/ to test rtmp:// cp67126.edgefcs.net/on%64emand/mp4:mediapm/ovp/content/test/video/spacealone%68d_sounas_640_300.mp4 (random letters in the app and playpath have been urlencoded) results in the characters being sent as-is: http://i.imgur.com/wBxp4.png Also note that the weird stream you're testing will accept spaces in addition to line breaks, so this whole urlencoding issue is easy to avoid for that stream: rtmpdump -r "rtmp:// freeview.fms.visionip.tv/live/tvnetwork-hellenictv-sigma-hsslive-25f-4x3-SDh" --live (In case the previous line gets broken up: there is a space between SDh and the quotation mark.)
On Wed, Oct 24, 2012 at 6:37 PM, NhJm wrote:
rtmp://cp67126.edgefcs.net/on%64emand/mp4:mediapm/ovp/content/test/video/spacealone%68d_sounas_640_300.mp4 (random letters in the app and playpath have been urlencoded)
You have a good example. This shows that some (if not most) servers will accept encoded characters. However I have a counter. Even if I use your clever "space" trick, the server in question will still reject other encoded characters. Notice I encoded the final "h" of the playpath, and that I used "-y" with "official" RtmpDump in order to avoid decoding for the purpose of this example. $ rtmpdump -o a.flv -r rtmp://freeview.fms.visionip.tv/live -y "tvnetwork-helle nictv-sigma-hsslive-25f-4x3-SD%68 " -V (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL DEBUG: Protocol : RTMP DEBUG: Hostname : freeview.fms.visionip.tv DEBUG: Port : 1935 DEBUG: Playpath : tvnetwork-hellenictv-sigma-hsslive-25f-4x3-SD%68 DEBUG: tcUrl : rtmp://freeview.fms.visionip.tv:1935/live DEBUG: app : live DEBUG: live : no DEBUG: timeout : 30 sec DEBUG: Setting buffer time to: 36000000ms Connecting ... DEBUG: RTMP_Connect1, ... connected, handshaking DEBUG: HandShake: Type Answer : 03 DEBUG: HandShake: Server Uptime : 96850052 DEBUG: HandShake: FMS Version : 3.0.1.1 DEBUG: HandShake: Handshaking finished.... DEBUG: RTMP_Connect1, handshaked DEBUG: Invoking connect INFO: Connected... DEBUG: HandleServerBW: server BW = 2500000 DEBUG: HandleClientBW: client BW = 2500000 2 DEBUG: HandleCtrl, received ctrl. type: 0, len: 6 DEBUG: HandleCtrl, Stream Begin 0 DEBUG: HandleChangeChunkSize, received: chunk size change to 4096 DEBUG: RTMP_ClientPacket, received: invoke 261 bytes DEBUG: (object begin) DEBUG: (object begin) DEBUG: Property: <Name: fmsVer, STRING: FMS/3,5,7,7009> DEBUG: Property: <Name: capabilities, NUMBER: 31.00> DEBUG: Property: <Name: mode, NUMBER: 1.00> DEBUG: (object end) DEBUG: (object begin) DEBUG: Property: <Name: level, STRING: status> DEBUG: Property: <Name: code, STRING: NetConnection.Connect.Su ccess> DEBUG: Property: <Name: description, STRING: Connection succeeded.> DEBUG: Property: <Name: data, OBJECT> DEBUG: (object begin) DEBUG: Property: <Name: version, STRING: 3,5,7,7009> DEBUG: (object end) DEBUG: Property: <Name: clientid, NUMBER: 1626554004.00> DEBUG: Property: <Name: objectEncoding, NUMBER: 0.00> DEBUG: (object end) DEBUG: (object end) DEBUG: HandleInvoke, server invoking <_result> DEBUG: HandleInvoke, received result for method call <connect> DEBUG: sending ctrl. type: 0x0003 DEBUG: Invoking createStream DEBUG: RTMP_ClientPacket, received: invoke 29 bytes DEBUG: (object begin) DEBUG: Property: NULL DEBUG: (object end) DEBUG: HandleInvoke, server invoking <_result> DEBUG: HandleInvoke, received result for method call <createStream> DEBUG: SendPlay, seekTime=0, stopTime=0, sending play: tvnetwork-hellenictv-sigm a-hsslive-25f-4x3-SD%68 DEBUG: Invoking play DEBUG: sending ctrl. type: 0x0003 DEBUG: HandleCtrl, received ctrl. type: 6, len: 6 DEBUG: HandleCtrl, Ping 8892 DEBUG: sending ctrl. type: 0x0007 DEBUG: HandleCtrl, received ctrl. type: 6, len: 6 DEBUG: HandleCtrl, Ping 17238 DEBUG: sending ctrl. type: 0x0007 DEBUG: HandleCtrl, received ctrl. type: 6, len: 6 DEBUG: HandleCtrl, Ping 25506
On Wed, Oct 24, 2012 at 7:29 PM, Steven Penny <svnpenn@gmail.com> wrote:
Even if I use your clever "space" trick, the server in question will still reject other encoded characters. Notice I encoded the final "h" of the playpath, and that I used "-y" with "official" RtmpDump in order to avoid decoding for the purpose of this example.
It is up to the server's code to parse URLs. It is the responsibility of the user to provide the server with URLs that the server can handle. The _official_ RTMP client does not decode URLs prior to sending them to the server, therefore rtmpdump should not be decoding URLs either.
$ rtmpdump -o a.flv -r rtmp://freeview.fms.visionip.tv/live -y "tvnetwork-helle nictv-sigma-hsslive-25f-4x3-SD%68 " -V
There's nothing we can do to fix the fact that the server (some version of Wowza, perhaps running some custom code) is not designed to handle urlencoding. (And I do mention "custom code" because the thought that Wowza itself requires whitespace at the end of a playpath scares me.)
On Wed, Oct 24, 2012 at 11:45 PM, NhJm wrote:
The _official_ RTMP client does not decode URLs prior to sending them to the server, therefore rtmpdump should not be decoding URLs either.
So then what does this do? Taken from official master parseurl.c if (*p == '%') { unsigned int c; sscanf(p+1, "%02x", &c); *destptr++ = c; pplen -= 3; p += 3;
On Thu, Oct 25, 2012 at 12:19 AM, Steven Penny <svnpenn@gmail.com> wrote:
The _official_ RTMP client does not decode URLs prior to sending them to
On Wed, Oct 24, 2012 at 11:45 PM, NhJm wrote: the
server, therefore rtmpdump should not be decoding URLs either.
So then what does this do? Taken from official master parseurl.c
if (*p == '%') { unsigned int c; sscanf(p+1, "%02x", &c); *destptr++ = c; pplen -= 3; p += 3;
It does do urldecoding. It would probably be best to remove it. (And if people need to include spaces in librtmp parameterized URLs, they can just do it the hard way by using the parameters.)
On Thu, Oct 25, 2012 at 12:35 AM, NhJm wrote:
It does do urldecoding. It would probably be best to remove it.
If we remove it from parseurl.c, then are we going to remove it from rtmpgw.c if (strncmp(data + src_x, "%", 1) == 0 && src_x + 2 < length) { // // Since we encountered a '%' we know this is an escaped character // hex[0] = data[src_x + 1]; hex[1] = data[src_x + 2]; data[dst_x] = (char) strtol(hex, &stp, 16); dst_x += 1; src_x += 3; } and rtmp.c if (*p1 == '\\') { unsigned int c; if (port < 3) return FALSE; sscanf(p1+1, "%02x", &c); *p2++ = c; port -= 3; p1 += 3; }
On Thu, Oct 25, 2012 at 12:47 AM, Steven Penny <svnpenn@gmail.com> wrote:
If we remove it from parseurl.c, then are we going to remove it from rtmpgw.c
The code in rtmpgw.c is to parse URL query parameters. (ex. http://localhost/?r=rtmp:%2F%2Ffoo%2Fblah&f=LNX%2010,0,32,18&y=asdf)
and rtmp.c
The code in rtmp.c is to parse parameters. It is clearly stated in the manual that parameters can be escaped using \xx. This is necessary if you want to include a space (\20) within a parameter value. (ie. "rtmp://foo/blah flashVer=LNX\2010,0,32,18 playpath=asdf")
Steven Penny wrote:
On Thu, Oct 25, 2012 at 12:35 AM, NhJm wrote:
It does do urldecoding. It would probably be best to remove it.
If we remove it from parseurl.c, then are we going to remove it from rtmpgw.c
No. These are not the same situation.
if (strncmp(data + src_x, "%", 1) == 0 && src_x + 2 < length) { // // Since we encountered a '%' we know this is an escaped character // hex[0] = data[src_x + 1]; hex[1] = data[src_x + 2]; data[dst_x] = (char) strtol(hex, &stp, 16); dst_x += 1; src_x += 3; }
and rtmp.c
if (*p1 == '\\') { unsigned int c; if (port < 3) return FALSE; sscanf(p1+1, "%02x", &c); *p2++ = c; port -= 3; p1 += 3; }
all i say are thanks xspeed1989 From: NhJm Date: 2012-10-24 12:39 To: rtmpdump discussion and development Subject: Re: [rtmpdump] [PATCH] Better URL decoding support On Tue, Oct 23, 2012 at 9:12 AM, Steven Penny <svnpenn@gmail.com> wrote: - /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - } Please be careful not to break things. Anyways, the proper behavior is to actually *NOT* do any urldecoding. The %xx should be sent to the server as-is. This is what the Flash Player does and is the same as how normal HTTP clients handle URLs.
NhJm wrote:
On Tue, Oct 23, 2012 at 9:12 AM, Steven Penny <svnpenn@gmail.com <mailto:svnpenn@gmail.com>> wrote:
- /* skip extension */ - if (subExt && p == ext) { - p += 4; - pplen -= 4; - continue; - }
Please be careful not to break things.
Anyways, the proper behavior is to actually *NOT* do any urldecoding. The %xx should be sent to the server as-is. This is what the Flash Player does and is the same as how normal HTTP clients handle URLs.
Correct. This patch should be discarded.
On Wed, Oct 24, 2012 at 8:54 AM, Howard Chu wrote:
Correct. This patch should be discarded.
I offered a response email to NhJm and it is ignored, and some rubbish from xspeed1989 is let through? You want to kill this project? Kill moderating the mailing list. You dont like my patch fine, but at least allow some discussion to take place, stop acting like a child. Otherwise I will simply upload the build anyway, 100 downloads a day on my "discarded" patches.
On Wed, 24 Oct 2012 09:36:15 -0500, Steven Penny wrote:
On Wed, Oct 24, 2012 at 8:54 AM, Howard Chu wrote:
Correct. This patch should be discarded.
I offered a response email to NhJm and it is ignored, and some rubbish from xspeed1989 is let through? You want to kill this project? Kill moderating the mailing list. You dont like my patch fine, but at least allow some discussion to take place, stop acting like a child. Otherwise I will simply upload the build anyway, 100 downloads a day on my "discarded" patches.
i undid your moderation flag, i was tired of manually approving your mails. not sure why hyc and you cant talk to each other? -compn
user in #mplayer reported this url does not work with git master http://www.vipboxsports.eu/baseball/92619/1/nlcs-game-7:-st-louis-cardinals-... works with stevens' patches. command: rtmpdump -r "rtmp://109.201.133.38:1935/redirect" -a "redirect" -f "WIN 11,4,402,287" -W "http://static.foxsportslatam.com/player.swf" -p "http://foxsportslatam.com/embed.php?id=2mlb1022_19830&width=660&height=470" -y "2mlb1022_19830" -o 2m.flv [21:21] <avocado> weird, i'm getting an error with it [21:21] <avocado> INFO: Connected... [21:21] <avocado> ERROR: rtmp server sent error [21:21] <avocado> ERROR: rtmp server requested close [21:21] <avocado> i just tried it with rmpdump from git [21:25] <Compn> https://github.com/svnpenn/rtmpdump [21:57] <avocado> Compn: thanks, that worked perfect of course its a live stream so the stream is gone now. -compn
On Wed, Oct 24, 2012 at 12:20 PM, compn wrote:
user in #mplayer reported this url does not work with git master http://www.vipboxsports.eu/baseball/92619/1/nlcs-game-7:-st-louis-cardinals-...
works with stevens' patches.
command:
rtmpdump -r "rtmp://109.201.133.38:1935/redirect" -a "redirect" -f "WIN 11,4,402,287" -W "http://static.foxsportslatam.com/player.swf" -p "http://foxsportslatam.com/embed.php?id=2mlb1022_19830&width=660&height=470" -y "2mlb1022_19830" -o 2m.flv
Yeah, its from the patch I committed from KSV patch 2 months ago http://lists.mplayerhq.hu/pipermail/rtmpdump/2012-September/002087.html Shot down horribly http://lists.mplayerhq.hu/pipermail/rtmpdump/2012-September/002088.html
participants (5)
-
compn -
Howard Chu -
NhJm -
Steven Penny -
xspeed1989