[FFmpeg-devel] [PATCH] Add NULL mixer to AVSequencer.

Sebastian Vater cdgs.basty
Mon Aug 16 08:22:42 CEST 2010

Ronald S. Bultje a ?crit :
> Hi,
> On Sun, Aug 15, 2010 at 12:05 PM, Sebastian Vater
> <cdgs.basty at googlemail.com> wrote:
>> Ronald S. Bultje a ?crit :
>>> On Sat, Aug 14, 2010 at 8:06 AM, Sebastian Vater
>>> <cdgs.basty at googlemail.com> wrote:
>>>> The NULL mixer does not output any PCM data but internally
>>>> simulates the whole mixing process (i.e. calculation of
>>>> sample positions and running the playback handler at
>>>> correct timing), which is useful for playtime calculation.
>>> For those of us that know less than you about mod files, can you
>>> explain in a few sentences what this patch is useful for? Why do we
>>> want it, why do we need it, why would it be important to commit this
>>> patch before - say - a demuxer/decoder, and what does the patch in the
>>> end accomplish, i.e. what is the place of this patch w.r.t. the big
>>> picture of ffplay playing back a mod file?
>> Consider it like /dev/null and /dev/zero. Although they don't do
>> anything "useful" they have their pretty places. ;-)
>> The purpose of the null mixer is:
>> a) exact playtime calculation. For TuComposer files you can't anymore
>> simply parse the pattern data and go through the order list, since it
>> features a full-featured synth sound assembler which makes it otherwise
>> impossible to get always correct results (unless you solve the famous
>> halting problem of a turing machine).
>> The point here is, that the synth assember can also call the normal
>> effects (that means also position jump and pattern changing, which
>> affect playback time). It also can read the current sample position of a
>> certain channel and do actions based on that, i.e. you can write code
>> with the synth sound assembler that does a pattern jump if current
>> sample position is between 7000 and 9000 (yes that's weird, but the
>> power is there).
> Apart from tucomp files, is this used? If not, is there a reason for
> that (e.g. needing a specialized mixer for that, which was too
> CPU-intensive on Amiga CPUs at that time)? Should we support it?

I really don't know, if someone else uses such fancy features. I know of
various very old tracker formats which are not a usual file format but
more like 68k code which directly plays the data. A demuxer could simply
convert this to synth sound assembly code.

Well, on the other hand, I just had another idea yesterday evening: seeking!

Normally, when you seek in a mod file, you seek one pattern forward or
backward but this causes problems, let's say we're at track 3, row 31
und we seek forward entering track 4 row 0. Now let's say there's a note
played at track 3, row 48 which would be ignored normally.

Here comes the null mixer into his role. On seeking forward we transfer
current channel data to the null mixer which is as easy as:
for (i = 0; i < num_channels; ++i) {
    get_channel ( lq_mixer, mix_data, i );
    set_channel ( null_mixer, mix_data, i );

The null mixer then continues mixing until seek position is reached.
Onto arrival it simply puts back the data to the low quality mixer and
continue playback.

For backward seeking we do similar, just that we reset the song (call
avseq_song_reset) and use the null mixer until seek position, then pass
the data to the low quality mixer again.

This will give exact seeking in the sense of audio output which is as
far as I know unique to FFmpeg then (for mod files), neat trick ;)

>> b) for transcoding from module to module when translate NNA channels to
>> non-NNA supporting trackers, the null mixer does it well enough to do
>> this correctly.
>> c) it's waaaay faster than even the low quality mixer, on my Amiga the
>> null mixer is around 10-30 times faster than the low quality mixer.
>> Imagine you are setting up an FFmpeg server which has PCM output
>> disabled for modules, then you can disable the low quality mixer and
>> just get stick with the null quality mixer. This is also useful for size
>> reduction, since the low quality mixer is around 450k (object file),
>> while the null mixer is only around 23k.
> This will never happen. I don't see FFmpeg having flags for disabling
> particular features of several obscure mod files.
> FFmpeg will have compilation flags to disable mod demuxers/decoders,
> or for disabling the synth mixer. That's it.

Well there will be actually more mixers, there will be a high quality
mixer also (if you wish I can send you two small TuComposer wav files,
one mixed with low quality and one with the high quality mixer, both at
4kHz, the difference is extreme!

The point is that the high quality mixer is also 20-50 times more CPU
intense than the low quality mixer.

Since the user can choose between the mixers, using the high quality
mixer for seeking would make that extreme slow (let the user wait quite
a time).

Also I plan to add SID chip mixing (for playback C64 files) and OPL2/3
AdLib FM synthesizer mixing (did you know that even S3M supports OPL2/3,
this is hard coded for channels 17-29 if I remember correctly, i.e. 1-16
are PCM, the rest OPL2/3). There are also other trackers being OPL2/3.
Maybe we also do a MIDI mixer (i.e. a mixer which interprets MIDI
commands and converts them using GUS patches and like that to PCM).

You see there are quite a lot of candidates for mixers, also a floating
point mixer might be done.

In fact any tracker and player handling mod files, esp. these today
support multiple mixers for varying targets.

Games and demos usually will utilize the low quality mixer (and disable
the high quality mixer as well as the null quality mixer, they don't
seek since in game music isn't supposed to be seekable). Players can use
all three if they wish, disk writers should use the high quality mixer,
as well.

It is even possible to change between the low and high quality mixer
during playback of one file. The application could start mixing with
high quality mixer and when it detects CPU load of over 50% it switches
to the low quality mixer and back when the number of channels decreases
again (somewhat like mp3 VBR).

>> d) It's good for review, since it's less than 1000 lines but these lines
>> share 95% with the low quality mixer, i.e. if this passes review, the
>> low quality mixer will most likely also (it's 23k lines now, but no
>> panic, most of the rest 22k lines are VERY similar, so just review in
>> the low quality mixer one or five functions which would follow after the
>> null mixer, and I'll apply the critics to the remaining functions, too,
>> saving us all LOTS of time).
> My impression is:
> - 23k source is too big. You have to simplify it.

I guess anyway that the code can be reduced to some 2-3k of lines by
using neat defines. As said most functions in the > 1k line range are
very similar and are simply specially optimized versions for 8, 16 and
32 bit samples, for mono, left, right, centered and surround panning.

> - 405k object file is unacceptable, but I think you took one with
> debug symbols, so let's revisit that with real numbers.

Well, the whole tucomposer.library in m68k 100% asm including low
quality mixer, IFF-TCM1 (de-)muxers, IFF-TCI1 instrument loader/saver,
etc. was 160kb in size without debug symbols, the C only version was in
total quite double the size. So I pretty guess from this that 405k is
because of the debug symbols. ;-)

Making it smaller by executable size would enormously slow it down.
Please note, saying that it can handle (depending on settings) 8-64
channels at 44,1 kHz on a 486 does not mean that you have any CPU power
free for other stuff, it's the hard limit before everything will freeze.
Also remember my post where I had the idea to not only PCM as input but
also directly mp3, ogg, etc. streams.

Having 64 channels with mp3s in parallel will still take quite some time
on todays CPUs.

After all mix_rate * number_of_channels calculations are required per

> - If null mixer and real mixer duplicate code, that duplication needs
> to be solved without duplicating it. If various functions in the
> 23k-source-mixer duplicate functionality, that should be prevented in
> a useful way also.

It's just now most duplicate, because I copied the low quality mixer
over to the null mixer and removed the non-skip mix routines, the rest
is pretty the same right now and I guess there can be a lot more
removed. So they will start to differ more.

Also there is another point, the low quality mixer allocates 256 * 256 *
sizeof (int32_t) bytes for 8-bit volume lut which the null mixer doesn't
have to.

Also I think that's consistent with FFmpeg to have a null mixer (don't
we also own a null avfilter graph?).

> We will not review one function and keep the design intact, if the
> design itself is flawed. My intention here is to come up with an
> optimal design that is better than anything any single person could
> have thought of by themselves, by evolving it to something better
> through review iterations.

Great idea! Multiple mixers, I really think, should be part of this.
But as always, let's start simple, and the null mixer is by far the
simplest, so... ;-)

> My question to you: can the null mixer problem be prevented by writing
> a smarter non-null-mixer that is faster when presented with a NULL
> output buffer? Does that lead to significantly smaller object file
> size with an insignificant effect on speed? Why / why not? If done
> correctly, can we live without the null mixer, thus only requiring a
> single synth mixer thus not needing a mixer interface, thus greatly
> simplifying the code that is to be submitted, thus making FFmpeg
> simpler and easier to maintain?

We should utilize multiple mixers any way. Since we want to target
multiple audiences, right?

If I remember correctly, even modplug has multiple mixers for various
goals. Using the NULL output buffer trick would require add checks all
over the place if it's not null, the null mixer makes this simply
transparent. Please note mix buffers are also used by trackers and
players for displaying nice graphics / equalizer stuff.


Best regards,
                   :-) Basty/CDGS (-:

More information about the ffmpeg-devel mailing list