[rtmpdump] Mysterious Invalid floating point operation in a modified RTMPDUMP

[Pepak]

Hi!

This doesn't concern RTMPDUMP directly, only a modification of it. But
I suspect the answer will be obvious to someone who is familiar with
both RTMPDUMP and the C language in general.

Using MingW, I am building the regular RtmpDump and a slightly
modified version of it (basically, rtmpdump.c modified to a DLL form,
where main() is exported (under a different name) and the argc/argv
variables come from the calling program). Outside of rtmpdump.c and
the makefile, nothing gets modified. Everything compiles without
any warnings. Up to version 2.2d, this DLL worked fine for me.

Since version 2.2e, up to and including the today's GIT version, the
DLL crashes while connecting stream. I located the problem to the
HandleInvoke function, specifically this line:

  txn = (int)AMFProp_GetNumber(AMF_GetProp(&obj, NULL, 1));

The problem is, during the fourth call of HandleInvoke,
AMFProp_GetNumber returns value 2147483653, which is too large to fit
into an int, and an exception is raised as a consequence. What I just
can't understand is, if I use RTMPDUMP itself (my compile) on the same
stream, it works just fine - it receives a *different* number , even
*larger* (2147509976), and apparently handles that just fine.

Unfortunately, my experience with C is limited, so I am rather baffled
as to why this happens. I would appreciate if someone could give me a
hint as to:

1) Why would I get a different number every time? Apparently if I run
RTMPDUMP a second time, I get yet another number (again larger than
MaxInt).

2) Why would range checking be used with my DLL but not with RTMPDUMP
itself? As far as I can tell, they both compile the same way:

rtmpdump: rtmpdump.o
        $(CC) $(LDFLAGS) -o $@$(EXT) $@.o $(LIBS)
rtmpdump_dll: rtmpdump_dll.o rtmpdump_dll.def
        $(CC) $(LDFLAGS) -s -shared $^ $> -o $@.dll $(LIBS)

3) How can I disable the range checking? I would prefer to do so
globally, as I suspect there might be a similar problem in other parts
of librtmp and I would rather prefer to "fix" them all at once than on
per-case basis.

4) This isn't really related, but still: When/if a solution is found,
should I submit the changes to the project itself? I think it might be
useful for non-C programmers to have RTMPDUMP as a library, but maybe
Howard would prefer to keep it separate?

Thanks for any suggestions.

Pepak