[FFmpeg-devel] headers for gapless playback

Sat Jan 18 01:14:33 EET 2020

Hey guys thanks for the discussion - sorry I went dark on this, I had a death in the family last November & am I the executor, so am just starting to free up again for projects. This is my first experience with ffmpeg coding, so I may have newbie questions. I just got a debug build completed & got a copy of the MP4 spec to research.  Issues so far are:

(Apologies in advance if I'm restating the obvious for anyone involved.) 
For AAC, the encoder can add blank samples to the input at either the beginning or end; their lingo for this is "encoder delay" and "encoder padding." For gapless playback, we'll need sample counts for the encoder delay and total pre-encoded samples. From what I've seen of the fdk-aac output, that's what they output. I'll look into the AV_PKT_DATA_SKIP_SAMPLES and side data as suggested.

As far as output, as I mentioned previously, adding the gapless info into the structure of a "moov" box seems to be the defacto standard most people are following in the players that I've seen so far. (I also see it's referred to as an ISO standard, but I haven't seen what standard specifically references gapless info. I'd be grateful for any further info on that if anyone has any.) I see there is already support in ffmpeg to emit "moov" boxes, and "edts/elst" boxes too, so I’m hoping I can use that & just figure out how to integrate the gapless data with existing.

As far as control, I'll need to add a config switch to control adding gapless data or not. In fdk-aac, they use the following (I'm proposing at first to do the "ISO standard" option):
-G, --gapless-mode <n> 
Encoder delay signaling for gapless playback
0	iTunSMPB (default)
1	ISO standard (edts + sgpd)
2	Both
There's also a switch to put the moov header before the sample data rather than at the end, which was a headscratcher for us when we researched this, since the players we worked with didn't work unless the moov header was first, and the MP4 spec says it's supposed to be first as well. I’m hoping ffmpeg already has this covered:
--moov-before-mdat 
Place moov box before mdat box on m4a output 
(source https://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC)
So at minimum I'd need some version of "-gapless".
I see "-g" is already taken, but would just plain "-gapless" be OK for first version?

That's all I've got so far. Will ping back when I get deeper into researching the code.
Thanks,
John

On 11/25/19, 3:00 PM, "ffmpeg-devel on behalf of Tomas Härdin" <ffmpeg-devel-bounces at ffmpeg.org on behalf of tjoppen at acc.umu.se> wrote:

    mån 2019-11-25 klockan 22:53 +0100 skrev Marton Balint:
    > 
    > On Sun, 24 Nov 2019, Tomas Härdin wrote:
    > 
    > > tor 2019-11-21 klockan 23:35 +0100 skrev Marton Balint:
    > > > On Thu, 21 Nov 2019, Tomas Härdin wrote:
    > > > 
    > > > > tis 2019-11-19 klockan 19:16 +0000 skrev John Kaplan:
    > > > > > Hi Guys,
    > > > > > 
    > > > > > We just did some research on gapless playback across multiple client
    > > > > > devices, and could not find support for adding gapless metadata
    > > > > > headers in ffmpeg. Our interest is specifically in AAC/MP4, and we
    > > > > > ended up shifting to the fdk-aac package, which exposes options for
    > > > > > adding moov/edts/elst headers (which are the ones we used) as well as
    > > > > > iTunSMPB.
    > > > > > 
    > > > > > So, I wanted to open a discussion with the team on ffmpeg support for
    > > > > > gapless metadata.
    > > > > > I’m not sure if you guys have discussed before, so here are a few
    > > > > > opening questions to get me up to speed…
    > > > > > 
    > > > > > 
    > > > > >   1.  Does ffmpeg support addition of gapless metadata in any form
    > > > > > currently, and we missed the doc as to how to specify it?
    > > > > >   2.  If not, have you discussed previously? If so, what were the
    > > > > > results? (I.e. rejected as evil, put on roadmap but in future,
    > > > > > currently in development, etc.)
    > > > > 
    > > > > Something very similar to this is being discussed in 
    > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__trac.ffmpeg.org_ticket_8366&d=DwIGaQ&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=i_WAJaOy8vrtjOFMFjd3LWqJsLBEpPDcwb8xZYhypuo&m=24yZeXB_Jb3Pva9bgKZgRPeQEdBbc112l5RqCB4cjyI&s=5KjCUx_qk8UI4w2shh8nPFGPsDda4UBFFwUvWO4QkJA&e=
    > > > > 
    > > > > >   3.  If support is still in the future, would you be open to me
    > > > > > contributing a patch that implements it?
    > > > > 
    > > > > Yes please
    > > > > 
    > > > > I think opening a generic ticket for this would be a good move. Having
    > > > > the ability to trim essence would be very useful. Then we could also
    > > > > remove that godawful hack in mov.c as well, hopefully. There's more
    > > > > than a few formats that need some "warm-up" data or that leave a few
    > > > > frames of garbage at the end of the file, like all block transform
    > > > > audio codecs
    > > > 
    > > > For audio we have AV_PKT_DATA_SKIP_SAMPLES packet side data
    > > 
    > > This is not enough for sample-level cutting as needed for gapless
    > > playback
    > 
    > Why not? The side data contains the number of samples to be skipped from 
    > the start and at the end of the packet.

    Aha, I missed that. That should work I guess. I still feel we should
    have an explicit "the actual media is between these timestamps"

    > > > for video AV_PKT_FLAG_DISCARD packet flag. What else is needed?
    > > 
    > > Why is this on the packets? It's not the demuxer's job to decide what
    > > should and should not be cut.
    > 
    > The demuxer only signals this, frames are dropped after they are decoded 
    > in generic libavcodec decode.c.

    That's OK at least. But see my previous comment.

    /Tomas

    _______________________________________________
    ffmpeg-devel mailing list
    ffmpeg-devel at ffmpeg.org
    https://urldefense.proofpoint.com/v2/url?u=https-3A__ffmpeg.org_mailman_listinfo_ffmpeg-2Ddevel&d=DwIGaQ&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=i_WAJaOy8vrtjOFMFjd3LWqJsLBEpPDcwb8xZYhypuo&m=24yZeXB_Jb3Pva9bgKZgRPeQEdBbc112l5RqCB4cjyI&s=Zjy6LfG45yBy4jt3BKA86CWcSAraVie4BZhBv4sOQYk&e=

    To unsubscribe, visit link above, or email
    ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".