[Libav-user] Using Libav to make HLS clips from H264

[Tyler Brooks]

Carl.
Thank you for your interest.
Sorry for the length of this note.

Short answer...
My encoder gives me timestamps but I don't know how to make multi-packet
frames that include the metadata.

Long answer...
My encoder gives me a timestamp on every frame (int64_t in microseconds).
The hardware is pretty good and the timestamps are very nearly always
separated to 1/framerate.  In my case the framerate is 30fps so the frames
come out of the encoder about every 30ms (lets just say for easy math for
now).

Of course, I could use those timestamps for my pts/dts.  Indeed, I have
tried.

But here is the root of my problem.  The key frames are handed to me as
four NAL packets (SPS, PPS, SEI, key_frame) all at the same time and all
with the same timestamp.  That makes sense.  The metadata meant for the
decoder (SPS, PPS and SEI) should to be associated with the leading
key_frame and they should all share the same timestamp.  *My problem is I
don't know how to make a multi-packet 'AVPacket' to submit as a single
frame to the 'av_write_frame' call. * Instead, I submit every packet as a
frame with no timestamp.  This seems to work.  Something inside the
'av_write_frame' call sees the metadata and squirrels it away for use by
the decoder.  Ultimately, I end up with a perfectly good HLS file(s).  But
unfortunately, that HLS file(s) has 'made up' timing.

If I use the same timestamp for all four pieces of the key_frame, then I
end up submitting packets as frames with the following timestamps (in this
case, I am generating a keyframe every 30 frames and my framerate is 30fps):

first SPS (pts/dts = 0ms)      // notice these all have the same timestamp
first PPS (pts/dts = 0ms)      //   .... and this one ...
first SEI (pts/dts = 0ms)        //                    ... and this one ...
first keyframe (pts/dts = 0ms)  //
  ... and this one
1st delta frame (pts/dts = 30ms)
2nd delta frame (pts/dts = 60ms)
3rd delta frame (pts/dts = 90ms)
...
29th delta frame (timestamp=970ms)
second SPS (pts/dts = 1000ms)     // notice these all have the same
timestamp
second PPS (pts/dts = 1000ms)     //   .... and this one ...
second SEI (pts/dts = 1000ms)      //                    ... and this one
...
second keyframe (pts/dts = 1000ms)  //
 ... and this one
1st delta frame (pts/dts = 1030ms)
2nd delta frame (pts/dts = 1060ms)
3rd delta frame (pts/dts = 1090ms)
...
29th delta frame (timestamp=1970ms)
...   and_so_on  ...

As you can see, I am telling the libraries that my frame timestamps are (in
order):

0, 0, 0, 0, 30, 60, 90 ... 970, 1000, 1000, 1000, 1000, 1030, 1060, 1090...
etc.

Those are *not* constantly increasing times stamps.  FFMPEG complains.

So, for now, I am marking every thing as unknown (AV_NOPTS_VALUE) which
produces an HLS file(s).

If I knew how to submit a multi-packet AVPacket to av_write_frame and
whatever was eating that packet saw the leading metadata and used it, then
I would be golden.

In general, am I going about this the right way?


On Mon, Nov 20, 2017 at 12:44 PM, Carl Eugen Hoyos <ceffmpeg at gmail.com>
wrote:

> 2017-11-20 17:50 GMT+01:00 Tyler Brooks <tylerjbrooks at gmail.com>:
> > I am using a Hi35xx camera processor from HiSilicon.  It is an Arm9 with
> a
> > video pipeline bolted on the side.  At one end of the pipeline is the
> CMOS
> > sensor.  At the other end is a H264 encoder.  When I turn on the
> pipeline,
> > the encoder outputs H264 NAL packets like this:
> >
> >     frame0: <SPS>,<PPS>,<SEI>,<key frame>
> >     frame1: <delta frame>
> >     frame2: <delta frame>
> >     ...
> >     frameN: <delta frame>
> >     frameN+1: <SPS>,<PPS>,<SEI><key frame>
> >     frameN+2: <delta frame>
> >     frameN+3: <delta frame>
> >     ...
> >     etc.
> >
> > I am turning that into HLS clips by doing the following (pseudo code for
> > clarity) :
> >
> >     av_register_all();
> >     avformat_network_init();
> >
> >     avformat_alloc_output_context2(&ctx_out, NULL, "hls", "./foo.m3u8");
> >
> >     strm_out = avformat_new_stream(ctx_out, NULL);
> >
> >     codec_out = strm_out->codecpar;
> >     codec_out->codec_id = AV_CODEC_ID_H264;
> >     codec_out->codec_type = AVMEDIA_TYPE_VIDEO;
> >     codec_out->width = encoder_width;
> >     codec_out->height = encoder_height;
> >     codec_out->bit_rate = encoder_bitrate;
> >     codec_out->codec_tag = 0;
> >
> >     avformat_write_header(ctx_out, NULL);
> >
> >     while(get_packet_from_pipeline_encoder(&encoder_packet)) {
> >       AVPacket pkt;
> >       av_init_packet(&pkt);
> >       pkt.stream_index = 0;
> >
> >       pkt.dts = AV_NOPTS_VALUE;
> >       pkt.pts = AV_NOPTS_VALUE;
> >       pkt.duration = (1000000/FRAMERATE);    // frame rate in
> microseconds
> >
> >       pkt.data = encoder_packet.data;
> >       pkt.size = encoder_packet.size;
> >
> >       if (is_keyframe(&encoder_packet)) {
> >         pkt.flags |= AV_PKT_FLAG_KEY;
> >       }
> >
> >       av_write_frame(ctx_out, &pkt);
> >     }
> >
> >     av_write_trailer(ctx_out);
> >     avformat_free_context(ctx_out);
> >
> > This seems to work fine except that the resulting HLS frame rate is not
> > right.  Of course, this happens because I am not setting the pts/dts
> stuff
> > correctly and ffmpeg lets me know that.  So I have two quetions:
> >
> >  1. Am I going about this right?
> >  2. How can I set the pts/dts stuff correctly?
> >
> > The encoder is giving me packets and I am submitting them as frames.
>
> > Those `<SPS>, <PPS> and <SEI>` packets are really out of band
> > data and don't really have a timestamp.
>
> If the encoder does not tell you the timestamps of the encoded
> frames, how are you - unrelated to FFmpeg ! - supposed to
> write files, no matter the format?
>
> Carl Eugen
> _______________________________________________
> Libav-user mailing list
> Libav-user at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/libav-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20171120/cbd61d1b/attachment.html>