[FFmpeg-user] Extract video frame data and frame metadata (PTS) to separate streams

[Rickard gräntzelius]

Hi!
I've been working on some image analysis using FFMPEG through python and I've ran in to what i think should be a solvable issue, but I can't figure out the approach/syntax.

The basic premise is the following:
I run through a video using the following command:

ffmpeg -ss _start_frame_ -i _file_ -f image2pipe -pix_fmt rgb24 -vcodec rawvideo -vsync 0 -frames:v _number_of_frames_ -filter:v fps=_framerate_ -

This gives me the frames at my required interval for analysis.

After analysis I want to save the relevant frames in my DB in a way that I can easily seek the same frame again from the source material.
This is where the problems start. If I just calculate based on starting frame and framerate the result I get from a seek operation does not match the frame I want to capture.

My first approach was to get PTS for every single frame from ffprobe, match it with a resultset of all the frames in the movie and then throwing away the surplus frames.
It works, but is way to slow.

I've looked in to matching my ffprobe command with the same parameters as the ffmpeg command, but it seems more limited and I didn't find a foolproof way.

A more elegant way would be to get the actual PTS of the frames I extract with the ffmpeg command.
I've seen that you can both burn the actual PTS value in the output frame, or put it in the filename.
However since I'm doing analysis on the frame I don't want to distort it, and I use the data directly from the pipe, so there's no filename involved.
I was hoping I could just output the PTS values in a separate data stream and I'm convinced it's possible, I just can't figure out how.

Any insights as to if this is actually possible, or the way forward is appreciated.

Ps.
Efficiency is a factor here.
We have another approach of iterating every frame explicitly from the start with multiple commands, and this works.
But letting ffmpeg iterate the video would improve performance a great deal.