[MPlayer-dev-eng] ao_oss: better handle SETFMT failure

[Alan Curry]

ao_oss has given me an unending stream of these:

[AO OSS] Can't set audio device /dev/dsp to s16be output, trying s16be...
[AO OSS] Can't set audio device /dev/dsp to s16be output, trying s16be...
[AO OSS] Can't set audio device /dev/dsp to s16be output, trying s16be...
...

Looking at http://manuals.opensound.com/developer/SNDCTL_DSP_SETFMT.html,
when SNDCTL_DSP_SETFMT changes the value of the format arg and returns 0, the
caller should use that new value as the format. mplayer ignores the new value
and tries signed 16-bit "native byte order". This is a guess that turns out
to be wrong when the sound driver doesn't have the same native byte order as
the CPU.

With the following patch, I get this more reasonable message:

[AO OSS] Can't set audio device /dev/dsp to s16be output, trying s16le...

And then things play nicely (presumably, mplayer is doing some byte swapping
somewhere).

Index: libao2/ao_oss.c
===================================================================
--- libao2/ao_oss.c	(revision 19127)
+++ libao2/ao_oss.c	(working copy)
@@ -215,6 +215,7 @@
 static int init(int rate,int channels,int format,int flags){
   char *mixer_channels [SOUND_MIXER_NRDEVICES] = SOUND_DEVICE_NAMES;
   int oss_format;
+  int err;
   char *mdev = mixer_device, *mchan = mixer_channel;
 
   mp_msg(MSGT_AO,MSGL_V,"ao2: %d Hz  %d chans  %s\n",rate,channels,
@@ -310,11 +311,18 @@
 #endif
     format=AF_FORMAT_S16_NE;
   }
-  if( ioctl(audio_fd, SNDCTL_DSP_SETFMT, &oss_format)<0 ||
+  if( (err=ioctl(audio_fd, SNDCTL_DSP_SETFMT, &oss_format))<0 ||
       oss_format != format2oss(format)) {
+    int failedformat=format;
+    if(err<0) {
+      format=oss2format(AF_FORMAT_S16_NE);
+    } else {
+      format=oss2format(oss_format);
+      if(format==-1)
+        format=oss2format(AF_FORMAT_S16_NE);
+    }
     mp_msg(MSGT_AO,MSGL_WARN, MSGTR_AO_OSS_CantSet, dsp,
-            af_fmt2str_short(format), af_fmt2str_short(AF_FORMAT_S16_NE) );
-    format=AF_FORMAT_S16_NE;
+            af_fmt2str_short(failedformat), af_fmt2str_short(format) );
     goto ac3_retry;
   }
 #if 0