[FFmpeg-devel] Splitting H.264 stream at key packet, first frame is not an I-frame

[Reimar Döffinger]

On 17.12.2013, at 23:29, Stefano Sabatini <stefasab at gmail.com> wrote:
> On date Tuesday 2013-12-17 20:55:57 +0100, Reimar Döffinger encoded:
>> On Tue, Dec 17, 2013 at 08:14:38PM +0100, Stefano Sabatini wrote:
>>> On date Wednesday 2013-12-18 02:56:34 +0900, Yusuke Nakamura encoded:
>>>> 2013/12/17 Stefano Sabatini <stefasab at gmail.com>
>>> [...]
>>>> For AVC and HEVC stream, IDR-picture or I-picture with
>>>> recovery_frame_cnt==0 themselves are not a Random Access Point (RAP).
>>>> The specification doesn't forbid that these pictures have no corresponding
>>>> parameter sets (VPS, SPS and PPS), but libavcodec/libavformat marks as a
>>>> keyframe even if the corresponding parameter sets are missing.
>>> 
>>> What I don't understand is how the VPS/SPS/PPS are processed in the
>>> FFmpeg framework, if they correspond to packets or if are stored
>>> somewhere else when processing the bitstream (e.g. as extradata
>>> used to decode the next frames).
>> 
> 
>> The first encountered is extracted into extradata, which means that
>> for the common case seeking and everything will work well even with
>> MPEG-TS.
>> However if there are additional, different ones they will only be
>> parsed as they come.
> 
>> The parser will ensure that they will be combined with the following
>> packet, which ideally should be an I-frame, thus splitting is supposed
>> to work just fine, too, both if either there is only one set of them
>> (and thus extradata contains the proper information) or if all I-frames
>> have them before.
> 
> In practice if there is no SPS/PPS before an I-frame, it is not merged
> with it, and in case the frame is chosen as the first frame of a new
> segment it will not contain the required information, and thus will
> not be decoded.

If the output stream does not start with a SPS/PPS the muxer should be generating them from extradata IMHO.
However out TS muxers are quite crappy, so it's no surprise if they don't.
What I said should be true if you mux into some other format like MP4 or MKV or ... (admittedly they have it easier and don't really support changing SPS etc...)

> Curiously, if I use the dump_extra bitstream filter I'm able to remove
> the playback error, but the first key frame is still missing:
> 
> $ ffmpeg -i IN.ts -bsf:v dump_extra -codec copy -f segment -map 0 -segment_time 6 seg-%d.ts
> 
> $ ffprobe seg-1.ts -show_entries packet=pts_time,flags:frame=pkt_pts_time,pict_type -select_streams v -of compact -read_intervals "%+#3"
> [...]
> Input #0, mpegts, from 'seg-1.ts':
>  Duration: 00:00:03.67, start: 6.286400, bitrate: 2628 kb/s
>  Program 1 
>    Metadata:
>      service_name    : Service01
>      service_provider: FFmpeg
>    Stream #0:0[0x100]: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p, 1024x576 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
>    Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 22050 Hz, stereo, fltp, 57 kb/s
> packet|pts_time=6.286400|flags=K
> packet|pts_time=6.328111|flags=_
> packet|pts_time=6.369811|flags=_
> frame|pkt_pts_time=6.328111|pict_type=P
> frame|pkt_pts_time=6.369811|pict_type=P

Huh? You should print the sizes, maybe the first is only PPS/SPS?
I doubt dump_extra does anything reasonable, unless it has specific support for converting H.264 extradata format?
Actually, most likely its broken format broke decoding of the I-frame, thus it is not output anymore and the following P-frames as a result are skipped as well (by default FFmpeg nowadays only starts decoding on the first I/keyframe nowadays).
Also, I only see that the noisy errors are gone, but I am unclear if those other frames are properly decoded (as said, I suspect they are actually just skipped).
Lastly, in principle it is possible to have P frames that decode stand-alone, though that would be a very strange case....