[FFmpeg-devel] Transcoding journey

[GOUJON Évan]

Hi everyone,

Let me first congrats all of you for this great project. Working on 
multimedia data is really challenging (math complexity, time 
constraints, broken formats) and you did it so you deserve many many 
thanks for that.

I'm currently developing an application that transcodes to several 
fomats depending on what the user sends to me.
I faced several small but really annoying issues so I wanted to share my 
journey.
I know the developer time is rather limited so maybe this is a perfect 
candidate for a GSoC cleanup project.
I apologize in advance for my long message.

My first issue was compiling examples (in doc/examples folder) because 
some of them are still using old and deprecated API.
For instance, transcoding.c still calls deprecated avcodec_encode_video2 
and avcodec_encode_audio2 functions.

For test purpose, I:
- generated a sample video using "ffmpeg -f lavfi -i 
testsrc=duration=5:size=1280x720:rate=2 testsrc.mp4"
- called "ffmpeg -loglevel debug -fdebug ts -i testsrc.mp4 testsrc.mkv" 
and debugged it in Eclipse until my own source code did the same.

I was surprised to find the first step was not creating the same file 
when called multiple times but I guess it's fine as long as the output 
is visually/timely the same.

Since my input data was in memory, I used avio_alloc_context with my own 
callbacks.
The function callback documentation is really poor and found out about 
AVSEEK_SIZE only after grepping the source code for its value.
I found using an int64_t offset a bit awkard since "size_t is a type 
guaranteed to hold any array index" but it's just a personal preference.

When opening the input, ffmpeg does use the "scan_all_pmts" option which 
is never specified in examples.
Turning it on by default may be a good idea but if it's a performance 
concern, that's ok.

Once opened, I count the number of video stream to ensure there is at 
least one.
I'm wondering if it's possible to have a container without any stream 
(for test-only purpose).
Note that when I open a picture file, the input AVStream 
"avg_frame_rate" is 0/0, not 0/1.

Configuring decoder and encoder parameters (framerate, pkt_timebase, 
sample_aspect_ratio, duration, side data) is a real pain and error-prone.
It would really be nice to have an helper to create an AVCodecContext 
from an AVStream.

I also discovered the output stream time_base is updated when calling 
avformat_write_header so calling it only when the encoder gives us the 
first frame is not a good idea.
I don't know if it's wanted but ffmpeg.c stores a muxer timebase 
(ost->mux_timebase) setting it to enc_ctx->time_base then "tr[ries] to 
improve muxing time_base" overwriting it to ost->st->time_base which 
only works because avformat_write_header has been called before. That 
seems very fragile to me.

ffmpeg.c does funky things like recomputing the output frame PTS using a 
float calculus (float_pts), resets pict_type to 0 (AV_PICTURE_TYPE_NONE) 
and even try to predict the final packet DTS using a complex approach 
using last packet duration, ticks, framerate and setting back and forth 
several variables pts, dts, next_pts, next_dts and last_ts.
I know this is used for statistics and the like but it's a real 
nightmare to follow.

After two weeks, I finally got a working video file but its duration was 
wrong by 0.5s (one frame).
I found the issue was in ff_compute_frame_duration because it uses the 
(deprecated) codec timebase.
It is filled in by ffmpeg.c which calls 
avcodec_copy_context(ost->st->codec, ost->enc_ctx) in init_output_stream 
function.
I tried replacing it with 
avcodec_parameters_from_context/avcodec_parameters_to_context as 
suggested by the avcodec.h avcodec_copy_context comment but the 
ost->st->codec->time_base is not updated so this is not equivalent.
I ended up copying the time_base field at hand but my code is now 
generating a deprecated warning.

I will try to submit a patch or two when I'll have time.
Anyway, thanks for reading me!