[FFmpeg-user] help remuxing mp4 to matroska

[Eric Gillum]

I am studying the source to see how command-line ffmpeg makes things
happen. Informative. I'll be trying things based on that. I'm still
wondering if there is more direct support for transmuxing...

On Thu, Sep 6, 2012 at 1:47 PM, Eric Gillum <ericwgillum at gmail.com> wrote:
> Hello,
>
> I am not an ffmpeg or multimedia expert. Bear with me. I am trying to
> transmux an mp4 file with h264 video and aac audio to matroska using
> libavformat, libavcodec, etc (i.e. not command line ffmpeg). My
> understanding is that no decoding or encoding is required. That is, I
> believe the matroska container can handle h264/aac, and that I have
> these bytes available, and therefore this should be more or less a
> straightforward copy. A quick sanity check of the file with ffprobe:
>
> Stream #0:0(und): Video: h264 (Baseline) (avc1 / 0x31637661)
> Stream #0:1(und): Audio: aac (mp4a / 0x6134706D)
>
> So first of all, what I guess must be transmuxing via command line is
> "ffmpeg -i 0.mp4 -vcodec copy -acodec copy 0.mkv". That gives me a
> playable file -- VLC can read and play it, and my player can read it
> (video plays back at least one frame and my implementation is hokey
> anyway; audio plays back fine).
>
> So I'm trying to transmux with libavformat / libavcodec. The code I
> use to transmux appears last in this message. Encouragingly, on
> opening the original file to transmux, I get these logs:
>
> [mov,mp4,m4a,3gp,3g2,mj2 @ 0x212fa00] Format mov,mp4,m4a,3gp,3g2,mj2
> probed with size=2048 and score=100
> [mov,mp4,m4a,3gp,3g2,mj2 @ 0x212fa00] ISO: File Type Major Brand: mp42
> [h264 @ 0x20f2600] err{or,}_recognition separate: 1; 1
> [h264 @ 0x20f2600] err{or,}_recognition combined: 1; 65537
> [aac @ 0x20e9400] err{or,}_recognition separate: 1; 1
> [aac @ 0x20e9400] err{or,}_recognition combined: 1; 65537
> [aac @ 0x20e9400] Unsupported bit depth: 0
>
> I get those same logs for the command-line-transmuxed files, so no
> obvious problem. And when transmuxing I get logs like these:
>
> [matroska @ 0x20fbe00] Writing block at offset 15, size 7349, pts 0,
> dts 0, duration 40, flags 128
> [matroska @ 0x20fbe00] Writing block at offset 21, size 4, pts 0, dts
> 0, duration 1024, flags 128
> [matroska @ 0x20fbe00] Writing block at offset 31, size 4395, pts 40,
> dts 40, duration 40, flags 0
> <snip>
> [matroska @ 0x20fbe00] Starting new cluster at offset 175 bytes, pts 1024
> [matroska @ 0x20fbe00] Writing block at offset 16, size 331, pts 1024,
> dts 1024, duration 1024, flags 128
> [matroska @ 0x20fbe00] Writing block at offset 354, size 2169, pts
> 1040, dts 1040, duration 40, flags 0
> <snip>
> [libx264 @ 0x2314800] final ratefactor: 29.14
>
> I get no warnings, no errors, etc. But the output file appears to be
> mostly junk. ffprobe detects the video dimensions, the audio sample
> rate, some scant codec stuff about h264 and aac...but generally
> information is reported as n/a. If I try to play this file, VLC does
> nothing and logs nothing. My own player appears not to be able to read
> the header. Here are logs when I open the output file:
>
> [matroska,webm @ 0x234ac00] Format matroska,webm probed with size=2048
> and score=100
> st:0 removing common factor 1000000 from timebase
> st:1 removing common factor 1000000 from timebase
> [h264 @ 0x234a800] err{or,}_recognition separate: 1; 1
> [h264 @ 0x234a800] err{or,}_recognition combined: 1; 65537
> [swscaler @ 0x6358000] No accelerated colorspace conversion found from
> yuv420p to rgba.
> [aac @ 0x2314200] err{or,}_recognition separate: 1; 1
> [aac @ 0x2314200] err{or,}_recognition combined: 1; 65537
> [aac @ 0x2314200] Unsupported bit depth: 0
> [h264 @ 0x234a800] Unknown NAL code: 25 (3009 bits)
> [h264 @ 0x234a800] no frame!
> [h264 @ 0x234a800] no frame!
>
> From my own logs (not shown) I can tell it reads very far into the
> file before reporting "Unknown NAL code", whereas on a good file it
> seems that my player gets a lot of what it needs from the first <4k in
> the file. As it is, this plays about 0.5 seconds of (correct) audio
> and then stops decoding due to errors. No video frames are decoded.
>
> So before I post the code, here are my questions:
> - Do I have the right idea about transmuxing from mp4 to matroska?
> - In the code you'll see that I do not ever call encode_video or
> decode_video. Perhaps I must still call these, and trust that no
> (expensive) reencoding occurs? How to I translate the "vcodec -copy"
> semantics?
> - Is there sample code or instructions on how to transmux with the
> libs? My search results only mention command line.
>
> And some last second details:
> - Built the libs from ffmpeg 0.9.1 (harmony) with debug logging.
> - Built with libx264.
> - The code below has been stripped of most comments, error checking,
> etc. I've already verified that no obvious error occurs. With the
> exception of my transmuxing code, my code runs and works. In fact,
> I've written code to encode in matroska format and then decode and
> play back. This is based on that, but basically with the encode_video
> and decode_video parts stripped out.
> - The code below omits IO, file processing, callbacks etc. Nothing
> interesting there (nothing ffmpeg-related).
>
> // Build a format context for writing (i.e. transmuxing).
> // ------------------------------------------------
> AVFormatContext *oc = avformat_alloc_context();
>
> // Matroska format.
> AVOutputFormat *fmt = av_guess_format("matroska", NULL, NULL);
> fmt->audio_codec = CODEC_ID_AAC;
> oc->oformat = fmt;
>
> // Make ffmpeg call me back when ready to write data.
> AVIOContext *ioctx =
> avio_alloc_context(av_malloc(CONTEXT_BUFFER_SIZE), /* buffer */
>                                         CONTEXT_BUFFER_SIZE, /* buffer size */
>                                         1, /* write flag */
>                                         opaque, /* opaque */
>                                         NULL, /* read packet */
>                                         callback, /* write packet */
>                                         NULL /* seek */);
> ioctx->seekable = 0;
> oc->pb = ioctx;
>
> // Add video stream.
> AVStream *video_st = avformat_new_stream(oc, NULL);
> AVCodecContext *c = video_st->codec;
> c->codec_id = CODEC_ID_H264;
> c->codec_type = AVMEDIA_TYPE_VIDEO;
> c->pix_fmt = PIX_FMT_YUV420P;
> c->width = 192;
> c->height = 144;
> c->time_base.den = frameRateForCurrentDevice();
> c->time_base.num = 1;
> c->bit_rate = 256000; // 512000
> c->bit_rate_tolerance = 128000; // 2 * c->bit_rate;
> c->refs = 4;
> c->gop_size = 4;
> c->max_b_frames = 4;
> c->me_range = 16;
> c->max_qdiff = 4;
> c->qmin = 10;
> c->qmax = 30;
> c->qcompress = 0.6;
> if (oc->oformat->flags & AVFMT_GLOBALHEADER) {
>     c->flags |= CODEC_FLAG_GLOBAL_HEADER;
> }
>
> // Open video codec.
> AVCodec *codec = avcodec_find_encoder(c->codec_id);
> avcodec_open2(c, codec, NULL);
>
> // Add audio stream.
> AVStream *audio_st = avformat_new_stream(oc, NULL);
> c = audio_st->codec;
> c->codec_id = CODEC_ID_AAC;
> c->codec_type = AVMEDIA_TYPE_AUDIO;
> c->sample_fmt = AV_SAMPLE_FMT_S16;
> c->bit_rate = 32000;
> c->bit_rate_tolerance = 16000;
> c->sample_rate = 44100;
> c->channels = 1;
> if (oc->oformat->flags & AVFMT_GLOBALHEADER) {
>     c->flags |= CODEC_FLAG_GLOBAL_HEADER;
> }
>
> // Open audio.
> codec = avcodec_find_encoder(c->codec_id);
> avcodec_open2(c, codec, NULL);
>
> avformat_write_header(oc, NULL);
>
> // Build a format context for reading, based on the file.
> // ------------------------------------------------
> char const *filename = "whatever";
> AVFormatContext *read_ctx = NULL;
> avformat_open_input(&read_ctx, filename, NULL, NULL);
> avformat_find_stream_info(read_ctx, NULL);
> // Find video stream.
> int read_video_stream_idx = -1;
> for (int i = 0; i < read_ctx->nb_streams; i++) {
>     if (read_ctx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
>         read_video_stream_idx = i;
>         break;
>     }
> }
> // Find audio stream.
> int read_audio_stream_idx = -1;
> for (int i = 0; i < read_ctx->nb_streams; i++) {
>     if (read_ctx->streams[i]->codec->codec_type == AVMEDIA_TYPE_AUDIO) {
>         read_audio_stream_idx = i;
>         break;
>     }
> }
>
> while (1) {
>     AVPacket pkt;
>     av_init_packet(&pkt);
>     int ret;
>     if ((ret = av_read_frame(read_ctx, &pkt)) < 0) {
>         break;
>     }
>
>     // Update stream index for writing.
>     pkt.stream_index = (pkt.stream_index == read_video_stream_idx) ?
> video_st->index : audio_st->index;
>     ret = av_interleaved_write_frame(oc, &pkt);
>     av_free_packet(&pkt);
> }
>
> avio_flush(oc->pb);
> av_write_trailer(oc);