[Libav-user] security camera app: bookmark+seek mp4 by wall-clock? low latency? visual timestamps?

[Alex Cohn]

On Fri, Aug 26, 2011 at 10:04, Camera Man <i.like.privacy.too at gmail.com> wrote:
> Dear List,
>
> (a) is there a general way to bookmark frames while writing a file in such a
> way that I can seek to them directly when playing, without searching? pts is
> NOT the answer, as explained below.

This is only relevant for key frames, but yes, you can. When writing
the stream, you can embed a copy of SPS/PPS preceeding every key
frame. Keep the file offset for this copy of SPS in your database, and
when you need it, you can simply start reading the file from this
offset.

> (b) is there a general way to encode *wall-clock* time  into a file I write?

I believe there isn't.

> (c) is there a way to force "low delay" handling of a stream despite its
> SPS/PPS description? for some cameras, video decoding is lagging 4 frames
> after the packet arrival, which - at 5fps, is almost 1 second delay. (one
> suggestion given below, looking for more options)

Why can't you fix the SPS to require low delay?

> (d) is there a way, other than subtitles, to add a visual timestamp to the
> file while writing it, without decoding+overlaying+reencoding?

What's wrong with using subtitles?

> (e) is there a way to tell, without decoding the video stream, that a
> received packet starts a new non-key frame?

Look at the first Nal code. It will most likely be 65 for I-frame, and
61 for P-frame.

BR,
Alex

> Discussion:
>
> for (a) (bookmarking), the solution that I am using so far is:
>
>  when recording, after I receive a packet from the RTSP stream, I note the
> exact (ntp synchronized) time, and the exact file offset using avio_tell(),
> and write them to a database, together with the AV_PKT_FLAG_KEY of the
> packet and other data. Every camera has its own output file.
>
>  when playing, to show a specific time, I independently in each file seek to
> the nearest preceding key frame using
> av_seek_frame(x,y,offset,AVSEEK_FLAG_BYTE), and run the following logic:
>
>  do {
>    av_read_frame(...);
>    avcodec_decode_video2(...);
>  } while (avio_tell(...) < file_offset_at_time_wanted);
>
>  it mostly works well for h264 "container" files (not really a container, as
> it has no header, footer or structure beyond the packets). However, if I try
> to do that in any structured file like avi,mp4,3gp, seeking by byte location
> does not seem to work properly (read_frame and or decode_video2 fail).
>
>  I'm looking for a solution that would work equally well for mp4 and avi
> files. It's possible that one does not exist -- for those formats that have
> a pts/dts, perhaps it is possible to use the pts/dts as index? (h264 files
> don't have pts or dts info at all)
>
> for (b) (wall clock), assume an mp4 or avi file; if I don't start pts/dts at
> 0, I get a delay at the beginning of the file (proportional to first frame's
> pts/dts); I suspect there is a way to mark a file "starting at a late pts",
> but I must have missed it in the docs? If I can do that, I can just
> av_seek_frame() by pts, relying on mp4/avi's frame index instead of my own.
> can I do that?
>
> for (c) (low latency), I found this post:
> <http://libav-users.943685.n4.nabble.com/Libav-user-latency-of-mpegts-handling-in-libavformat-tp3678681p3695084.html>
> with a suggestion for a solution. Seems to work, but looks fragile, and
> requires 4 calls to decode for each read_frame. Perhaps there is a better
> way?
>
> for (d) (visual timestamp) and (e) (frame boundary) I have no idea.
>
> Thank you for your time and ideas,
>
> Camera Man.