[MPlayer-dev-eng] amd64 -sws 0 fix (iow: playing with NX bit)

[Michael Niedermayer]

Hi

On Thursday 20 January 2005 23:08, Aurelien Jacobs wrote:
> Hi,
>
> Reimar pointed out a bug in swscaler introduced by my amd64 patch.
> This leaded me to do some sws tests on amd64. And I found out that
> upscaling with -sws 0 crashed. There was 2 reason for this. First
> there was bugs in my registers handling. The attached patch fix
> this, and I will commit it soon.
>
> The second reason is more problematic. This scaler use some code
> which is generated at runtime (the funnyCode). The generated code
> is simply written in a char[].
> Now, remember we are speeking of amd64. And remember that shiny
> new feature introduced by amd64, which is called NX bit....
> Yes, you've got it. The CPU simply prevent the execution of
> this code which is in a data segment, causing a segfault.
>
> The only way I found to be able to use this, is to boot the kernel
> with noexec=off parameter, thus disabling the use of this NX bit.
> Theoricaly another solution would be to use `execstack -s mplayer`
> to teach the kernel that it should not use the NX bit for this
> program, but it didn't worked for me.
>
> Now the question is how to solve this, cleanly ?
> I can see few options :
> (1) Find how to make execstack working (or any other way to set
>     executable stack flag, it seems that the linker is also able
>     to do it).
> (2) Rewrite this code so that it don't need to be generated at
>     runtime. This would be clean but this would be slower and
>     thus is not acceptable.
> (3) Simply document that -sws 0 need to disable NX bit support on
>     CPUs which have it.
> (4) Runtime detection of the NX bit, and clean exit with a
>     clear message if -sws 0 is used.
> (5) Probably other solutions, maybe better...
>
> What do you think about this ?

isnt it possible to switch the NX bit before and after executing the code 
somehow?

[...]
-- 
Michael

"nothing is evil in the beginning. Even Sauron was not so." -- Elrond