[FFmpeg-devel] [PATCH] configure: fix the bigendian test

[David Michael]

On Wed, Nov 11, 2020 at 6:20 PM Carl Eugen Hoyos <ceffmpeg at gmail.com> wrote:
> Am Mi., 11. Nov. 2020 um 16:30 Uhr schrieb David Michael <fedora.dm0 at gmail.com>:
> > There are two issues:
> >
> > The unused global integer does not make it into the compiled object
> > file, so declare it static to correct this.
>
> How can I reproduce this issue?

The configure test is basically these lines (where I use a PowerPC GCC
for a big-endian compiler):

echo "unsigned int endian = 'B' << 24 | 'I' << 16 | 'G' << 8 | 'E';" |
powerpc-gentoo-linux-gnu-gcc $CFLAGS -c -o test.o -xc - &&
od -A n -t x1 test.o | tr -d '\n' | grep -o '42 *49 *47 *45'

In testing this just now, it seems that the static value is only
present when "-O2 -g" is in CFLAGS, which I was always using as the
bare minimum flags from various distro packaging environments.  So
without static, the test is broken with "-flto"; with static, the test
is broken without "-O2 -g".  Maybe it could just check for a
_BIG_ENDIAN preprocessor definition instead.

> > The hexdump output is line-wrapped, so the expected value will not
> > be matched when its four bytes are split across two lines.  Use the
> > POSIX "-A n" option to disable printing input offsets and delete
> > all newline characters to output continuous hex values to grep.
>
> Will this work on AIX?

I don't have access to test it, but the behavior is described in the
Open Group specification, so I assume it would.

Thanks.

David