[FFmpeg-devel] questions on live streaming of h.264 in a FLV container

[Raymond Peck]

Hey there!

We have a system which streams live video which is encoded into an FLV
container using ffmpeg.  Playback is through a Flash player.  The
streaming server ensures that the player gets data which begins on a
keyframe (not necessarily on an IDR frame. . .) by throwing away video
stream data until it sees a keyframe.

I'm trying to get it to work with libx264 instead of the other codec.
When I start the player so that it's listening to an empty pipe, and
then start the video stream, the player plays the x264/FLV video just
fine.  This is very encouraging.

However, when the player comes in in the middle of the steam I get no
video.  I'd love to hear any clues that anyone might have, especially
if they have a similar system running.  :-)

I see in the flvenc.c code that the "extradata", which contains the
SPS and PPS data, is sent only at the start of the output file.  I
understand from an overview of h.264 that "Special NAL unit types are
specified to setup (sic) and change SPS and PPS in-band".  I've tried
various things to inject the extradata before keyframes to try to get
the player to pick up the SPS/PPS.  See below.

So, questions are:

1. It seems that the player would require an IDR frame to start, but I
haven't yet found out how to easily detect an IDR frame.  Does this
seem necessary or can I start on any keyframe?  I believe I'm
currently not outputting B-frames, so perhaps x264 is making every
keyframe an IDR frame?  Or am I getting P-frames which predict from
frames that preceed the previous keyframe?

2. Is it in fact possible to re-inject the SPS/PPS data in a similar
way to how the code surrounding ff_isom_write_avcc() does, and expect
the player to pick it up?  The "in-band" comment above would seem to
indicate so. . .

3. Do I instead need to re-create the header like flv_write_header()
does, when the player connects?  If so, then injecting the extradata
on the keyframes will allow the web server to extract and then reinject
the data as needed. . .


The code that I have now replicates the header code as follows and
calls it before outputting keyframes in flvenc.c.  Note that I've
changed the VIDEODATA.FrameType from 1 (keyframe) to 5 (video
info/command frame).  Before doing this, VLC paused on every keyframe
when playing a static file with this extra data; now it plays
smoothly.


static int flv_write_sps_pps_video_atom(AVFormatContext *s, int
timestamp, int timestamp_ext, int32_t composition_time) {
    ByteIOContext *pb = s->pb;
    AVCodecContext *enc = video_enc;

    if (enc->codec_id == CODEC_ID_AAC || enc->codec_id == CODEC_ID_H264) {
	int data_size;
	int streamid = 0;

	fprintf(stderr, "adding AVCDecoderConfigurationRecord atom with ts
%d, composition_time %d\n", timestamp, composition_time);

	offset_t pos;
	put_byte(pb, enc->codec_type == CODEC_TYPE_VIDEO ?
		 FLV_TAG_TYPE_VIDEO : FLV_TAG_TYPE_AUDIO);
	put_be24(pb, 0); // size patched later
	put_be24(pb, timestamp); // ts
	put_byte(pb, timestamp_ext); // ts ext
	put_be24(pb, streamid); // streamid
	pos = url_ftell(pb);
	if (enc->codec_id == CODEC_ID_AAC) {
	    assert(0);
	} else {
// this is what the header code does, but probably isn't what we want here:
//	    put_byte(pb, enc->codec_tag | FLV_FRAME_KEY); // flags
// instead, the FrameType should be "video info/command frame"
	    put_byte(pb, 5); // flags == video info/command frame
	    put_byte(pb, 7); // VIDEODATA CodecID = 7 (avc)
	    put_byte(pb, 0); // AVC sequence header (we hope that this is an
AVCDecoderConfigurationRecord, NOT an NALU tag)
	    put_be24(pb, composition_time); // composition time
	    ff_isom_write_avcc(pb, enc->extradata, enc->extradata_size);
	}
	data_size = url_ftell(pb) - pos;
	url_fseek(pb, -data_size - 10, SEEK_CUR);
	put_be24(pb, data_size);
	url_fseek(pb, data_size + 10 - 3, SEEK_CUR);
	put_be32(pb, data_size + 11); // previous tag size
    }

}


I'm very new to video encoding and to the ffmpeg internals and h.264,
so any clues are appreciated!

Thanks!