[Ffmpeg-devel] [RFC] Real demuxer and audio reordering

[Roberto Togni]

Hi all.

Attached you can find a patch to move audio data reordering in
rm demuxer for cook and 28_8 decoders. It also fix the extradata
problem by storing it with a fixed endian (i chose BE because all
data in rm files is BE).
It's implemented to send data to the codecs in the same way as realplayer.


There are a few open points i'd like to discuss before creating the
final patch.


*** Definitions
There are 4 numbers that need to be considered:
block_align: the size of a packet read from file
sub_packet_h: the number of packets in a block
coded_framesize: the size of the minumal decodeable unit (not for cook)
sub_packet_size: the size of the minimal decodeable unit (for cook only)

for cook decoder a packet of coded_framesize size can be further
splitted in subpackets of size sub_packet_size.

A block is a set of sub_packet_h packets, each of them with size
block_align.


*** How things works now (but they don't)
The demuxer reads a packet from the file, and sends it to the decoder.
The demuxer doesn't know anything about blocks and reordering.

Pro: simple, no codec-dependent code in the demuxer (about reordering,
there is codec-dependent code at other places)

Cons: the codec needs to buffer data and perform reordering, and then
returns a huge block of decoded audio; the codec does not know about
seeking or packet loss, so there is risk of desync; the codec needs a
lot of container data to perform these operations; if we want to use
the binary decoder we need to implement reordering before sending out
the data


*** How things works with this patch
now (with this patch applied): the demuxer collects a block of N audio
packets from the file (starting at an audio keyframe), apply the
reordering algorithm to the data, and then returns packets from the
reordered buffers when requested. When buffer is empty the process
starts again.

Pro: it works, it works also with binary codecs, the number of packes
read from file and sent to the codec is the same (no idea if that's
useful for some applications), the codec is called a fewer times with
bigger blocks of data.

Cons: the codec needs to split the packets into subpackets of
coded_framesize (and more in case of cook), and to do it it needs some
data from the container like codec_framesize and/or sub_packet_size


*** How things could work
The demuxer does buffering and reorder like above, but instead of
sending out packet of size block_align, it sends out packet of size
coded_framesize (or sub_packet_size for cook).

Pro: no splitting is done in the codec, no container data (except from
regular codec extradata) needs to be passed to the codec; should work
with binary codecs (unchecked)

Cons: for some codec the coded_framesize is quite small, so the codec
is called a lot of times; there are many output packets for every
packet got from file; we must create timestamps for intermediate
packets (but see below)


*** Extradata
Now we pass two kind of extradata to the codec: the fist part is made
from container data, the second part (if exists) is the "good" codec
extradata.

The endianness problem is easily fixed by forcing a known endianness to
the extradata fields (BE is the obvious choice in this case, see above)

The amount of extra extradata that is needed by the codec depends on
the packetization method, and on the level of compatibility we want to
keep with the binary codecs.


*** Timestamps
In all samples i tested, all packets in a block have the same
timestamp. In the patch i store them anyway and put them on the
reordered packets, but this is probably not needed.
Is it ok to have many packets with the same timestamp? I don't see any
problem in my test, and the current behaviour (without my patch) is the
same.


*** Notes on the patch
- lavf real demuxer is broken with all the 28_8 samples i tested (old
and new fileformat).
- i implemented reordering only for codec supported by lavc
- reordering 28_8 in old .ra format is not supported yet (it doesn't
work now anyway)
- 14_4, dnet (and atrac, even if not supported) does not require any
reordering
- raac (aac in real files) require a different kind of packet
splitting, and is not supported even now
- Moving reordering to the demuxer requires that the muxer performs
"disordering" when copying audio to a rm container (does it work
at all now? is the container data preserved?).
- The codec part needs cleanup, it was done with minimal changes now
- rm.c.diff is against cvs file, cook.c is against posted version


*** Final questions
- Is anybody against moving reordering to the demuxer?
- Does anybody (outside ffmpeg/ffplay) uses lavf rm demuxer?
- The change breaks compatibility for 28_8 codec, any complaint about
it? Is anybody using it now (MPlayer have its own copy because of
block size incompatibilities)
- Do we care about being able to use binary codecs with lavf demuxer?
- Any comment on the different solutions?

If nobody comments/gives better solutions i'll go on with the strategy
of this patch and keep all extradata to mantain binary compatibility.


Ciao,
 Roberto
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rm.c.diff
Type: text/x-patch
Size: 7402 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20051120/1c0cd9a8/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cook.c.diff
Type: text/x-patch
Size: 2647 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20051120/1c0cd9a8/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ra288.c.diff
Type: text/x-patch
Size: 1330 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20051120/1c0cd9a8/attachment-0002.bin>