[FFmpeg-devel] MOD support for FFmpeg (My GSoC 2010 task starts tomorrow)

[Vitor Sessak]

On 05/27/2010 03:35 PM, Ronald S. Bultje wrote:
> Hi,
>
> On Thu, May 27, 2010 at 7:56 AM, Peter Ross<pross at xvid.org>  wrote:
>> On Thu, May 27, 2010 at 04:31:59AM +0200, Michael Niedermayer wrote:
>>> On Sun, May 23, 2010 at 11:27:37PM +0200, Sebastian Vater wrote:
>>> I dont think the 1. and. 2. are exclusive, it surely should be possible
>>> to have a seperate lib and at the same time provide "mod" decoding support
>>> through the existing APIs.
>>> And we need to support it through existing apis because anything else would
>>> be quite inconvenient for applications.
>>> I am a bit undecided though about seperate lib or no seperate lib
>>
>> I toyed with ProTracker playback in FFmpeg last year, but misplaced the code
>> and gave up. My approach was a hybrid between the ones Sebastian listed, namely:
>>
>> * treat a module as an AVMEDIA_TYPE_AUDIO stream containing CODEC_TYPE_PATTERN.
>>
>> * add data structure(s) to AVCodecContext that describe the contents of a
>> sound pattern/module file. The structure must support concepts likes Patterns,
>> Notes, and Special Commands, etc. (TuComposer will have this already.)
>>
>> * write demuxers for various module formats. The demuxer would read the file
>> and converts it into the AVCodecContext pattern data structure.
>>
>> * write a libavcodec decoder called 'Pattern decoder', that takes CODEC_TYPE_PATTERN
>> as input. The decoder will process the AVCodecContext structure and outputs audio
>> samples (SAMPLE_FMT_xxx). Esentially this codec would invoke the TuComposer
>> functionality.
>>
>> * write muxers. thus allowing conversion between alike-module formats.
>
> I had something similar in mind, we (Vitor, me and Sebastian)
> discussed this on IRC earlier. However, we'd like to more-or-less keep
> the mod format even in "decoded" samples.
>
> - write demuxers that can "parse" mod files. These would be small,
> like raw demuxers. The output is AVMEDIA_TYPE_AUDIO with
> CODEC_TYPE_S3M/XYZ etc.
> - write decoders that can "decode" mod file b{it,yte}streams into
> collections of notes, etc. The output would be SAMPLE_FMT_MOD or
> SAMPLE_FMT_COMPOSER. Mans wants it to be called SAMPLE_FMT_KITTEN.

I'm still undecided of what is best, AVMEDIA_TYPE_AUDIO + 
SAMPLE_FMT_COMPOSER or AVMEDIA_TYPE_COMPOSER. But the argument for 
AVMEDIA_TYPE_AUDIO of having audioconvert.c do the PCM conversion in a 
way that is transparent for caller apps is appealing.

> - the "tracker" would then be a converter in the style of
> audioconvert.c, which converts from SAMPLE_FMT_KITTY to
> SAMPLE_FMT_S16/U8/FLOAT

> - a FIXME here's is how a user would choose the samplerate, maybe a
> AVCodecContext.request_sample_rate in the style of channel_mask?

Maybe "-sampler_flags" in the same way we have "-sws_flags"?

> - Vitor had another bunch of comments here but I forgot what it was
> exactly. Vitor?

Yes, I'd say that a very important piece of code would be the data 
structures holding the decoded collection of notes, instruments, etc. 
This will be what the decoders will output (and what we mean by 
SAMPLE_FMT_COMPOSER). Since this should be able to describe all 
supported MOD formats, it should be pretty flexible. I think it should 
be part of libavcodec. Also, the caller app might want to known which 
instruments are being played and when and how, so this structs should be 
part of the public API (which implies tough review). IMHO, this should 
be the first thing to be reviewed and committed, since it blocks all the 
rest.

> The advantage of this approach is that we can render between mod
> formats without complete re-encoding, we only have to reformat the
> b{it,yte}stream from the note collections.

At least for the conversion from a format with less supported features 
for one with more it looks pretty easy. I suggest getting it to work 
before starting sending the patches to the SAMPLE -> PCM conversion.

> We've suggested that Sebastian starts with #1, then moves to #2. For
> #3 he could write a tucomposer wrapper to start with, but eventually
> we'd want something in FFmpeg. He can start from tucomp code
> where-ever he wants, but the code will eventually live in FFmpeg SVN
> and thus go through full review here. He'll probably need a ff-soc
> repo account to put his work-in-progress because keeping track of 100
> patches will make us dizzy.

Every soc student should have a soc repo account...

-Vitor