[rtmpdump] Mysterious Invalid floating point operation in a modified RTMPDUMP

[Pepak]

Well, I did a search and found that the casting from AMFProp_GetNumber
to int is only performed in two places. So I wrote a simple function
which first tests the double value and only if it falls within the
integer ranger casts it; otherwise it returns MIN_INT right away.
Could this please be added to the official sources? It shouldn't hurt
anything in RTMPDUMP itself.

All changes are limited to rtmp.c

1. add
#include <limits.h>

2. add
int DoubleToInt(double n) {
        return ((n < INT_MIN) || (n > INT_MAX)) ? INT_MIN : (int) n;
}

3. Search for (int)AMFProp_GetNumber and replace it with
DoubleToInt(AMFProp_GetNumber. (Don't forget to add the closing
parenthesis at the end). In the 2011-07-20 source only two instances
of this call exist.

Thanks,

Pepak

> Hi!

> This doesn't concern RTMPDUMP directly, only a modification of it. But
> I suspect the answer will be obvious to someone who is familiar with
> both RTMPDUMP and the C language in general.

> Using MingW, I am building the regular RtmpDump and a slightly
> modified version of it (basically, rtmpdump.c modified to a DLL form,
> where main() is exported (under a different name) and the argc/argv
> variables come from the calling program). Outside of rtmpdump.c and
> the makefile, nothing gets modified. Everything compiles without
> any warnings. Up to version 2.2d, this DLL worked fine for me.

> Since version 2.2e, up to and including the today's GIT version, the
> DLL crashes while connecting stream. I located the problem to the
> HandleInvoke function, specifically this line:

>   txn = (int)AMFProp_GetNumber(AMF_GetProp(&obj, NULL, 1));

> The problem is, during the fourth call of HandleInvoke,
> AMFProp_GetNumber returns value 2147483653, which is too large to fit
> into an int, and an exception is raised as a consequence. What I just
> can't understand is, if I use RTMPDUMP itself (my compile) on the same
> stream, it works just fine - it receives a *different* number , even
> *larger* (2147509976), and apparently handles that just fine.

> Unfortunately, my experience with C is limited, so I am rather baffled
> as to why this happens. I would appreciate if someone could give me a
> hint as to:

> 1) Why would I get a different number every time? Apparently if I run
> RTMPDUMP a second time, I get yet another number (again larger than
> MaxInt).

> 2) Why would range checking be used with my DLL but not with RTMPDUMP
> itself? As far as I can tell, they both compile the same way:

> rtmpdump: rtmpdump.o
>         $(CC) $(LDFLAGS) -o $@$(EXT) $@.o $(LIBS)
> rtmpdump_dll: rtmpdump_dll.o rtmpdump_dll.def
>         $(CC) $(LDFLAGS) -s -shared $^ $> -o $@.dll $(LIBS)

> 3) How can I disable the range checking? I would prefer to do so
> globally, as I suspect there might be a similar problem in other parts
> of librtmp and I would rather prefer to "fix" them all at once than on
> per-case basis.

> 4) This isn't really related, but still: When/if a solution is found,
> should I submit the changes to the project itself? I think it might be
> useful for non-C programmers to have RTMPDUMP as a library, but maybe
> Howard would prefer to keep it separate?

> Thanks for any suggestions.

> Pepak

> _______________________________________________
> rtmpdump mailing list
> rtmpdump at mplayerhq.hu
> https://lists.mplayerhq.hu/mailman/listinfo/rtmpdump