[FFmpeg-devel] [PATCH] mxfdec: make it work with other calling conventions

[Reimar Döffinger]

On Wed, Jun 30, 2010 at 05:05:14PM -0700, Baptiste Coudurier wrote:
> On 06/30/2010 04:50 PM, Reimar D?ffinger wrote:
> >On Wed, Jun 30, 2010 at 04:20:50PM -0700, Baptiste Coudurier wrote:
> >>On 06/30/2010 04:06 PM, Reimar D?ffinger wrote:
> >>>On Wed, Jun 30, 2010 at 02:49:55PM -0700, Baptiste Coudurier wrote:
> >>>>>>But that's more an explanation of
> >>>>>>why there isn't any prototype for the read() function.
> >>>>>
> >>>>>I expected that is because pointers to functions with a different
> >>>>>number of arguments were stored there, but that is more of an argument
> >>>>>_for_ making it an error, since this allowed the fact that the
> >>>>>code would crash with stdcall and all other calle-restores-stack calling
> >>>>>conventions to go by unnoticed, without even a warning.
> >>>>
> >>>>All the functions are called with the same number of arguments.
> >>>
> >>>That's completely irrelevant, they are called with a different number
> >>>of arguments than the number of arguments they are declared with!
> >>
> >>In any case, the compiler has to fail in this case, because
> >>vararg/undefined prototype cannot work and stdcall requires a
> >>function prototype.
> >
> >What? Whatever gave you that idea?
> >Microsoft all those years has never noticed that none of their sofware
> >can even compile?
> 
> http://msdn.microsoft.com/en-us/library/zxk0tw93(VS.71).aspx
> 
> The __stdcall calling convention is used to call Win32 API
> functions. The callee cleans the stack, so the compiler makes vararg
> functions __cdecl. Functions that use this calling convention
> require a function prototype.

Well, they never bothered to implement varargs for stdcall, though
it wouldn't be impossible.
But I have no idea what they are talking about with the prototype.
I expect they mean at the point where the function is defined,
certainly not where it is used.
Simple test-case with mingw gcc:
id __stdcall test2();

int __stdcall test(int a)
{
    test2(0);
    test2(0ULL);
    return 1;
}

gcc happily generates code that is guaranteed to crash:
   0:   55                      push   %ebp
   1:   89 e5                   mov    %esp,%ebp
   3:   83 ec 18                sub    $0x18,%esp
   6:   c7 04 24 00 00 00 00    movl   $0x0,(%esp)
   d:   e8 00 00 00 00          call   12 <_test at 4+0x12>
  12:   52                      push   %edx
  13:   c7 04 24 00 00 00 00    movl   $0x0,(%esp)
  1a:   c7 44 24 04 00 00 00    movl   $0x0,0x4(%esp)
  21:   00 
  22:   e8 00 00 00 00          call   27 <_test at 4+0x27>
  27:   50                      push   %eax
  28:   50                      push   %eax
  29:   b8 01 00 00 00          mov    $0x1,%eax
  2e:   c9                      leave  
  2f:   c2 04 00                ret    $0x4

(interesting how it does not use push to store the
arguments but instead uses it to un-cleanup the stack...
I'm almost speechless).