[FFmpeg-devel] Custom Video Encoding and Decoding

[Jaeho Bang]

Dear Paul,
Thank you for the reply! I must have done a bad job explaining, sorry about
that. There is some underlying information that I have left out. My initial
clustering algorithm / representative frame selection algorithm actually
spits out a frame importance hierarchy. I want to implement a custom
decoder because it needs to process inputs such as 'select 100 most
important frames of this video'. Rather than decoding the entire video and
choosing the 100 most important frames based on frame importance hierarchy,
I want to only decode the frames that I specify (By having a custom
encoder, I have already forced these frames to be i-frames, hence minimum
computation is required). The goal of this decoder is to minimize memory
footprint (since decoding entire video into image frames increases memory
usage by 100x according to my experiments) while maximizing decoding speed.
Does this provide a clearer reason why I am thinking of implementing a
custom decoder? Please feel free to ask for clarifications.

Thank you,
Jaeho Bang

On Wed, Jul 29, 2020 at 3:31 PM Paul B Mahol <onemda at gmail.com> wrote:

> On 7/29/20, Jaeho Bang <jaeho.bang at gmail.com> wrote:
> > Dear members of ffmpeg-devel,
> >
> > I am currently working on a research project and from looking over the
> > examples and documentation in the ffmpeg repository, I believe I need to
> > implement a custom encoder / decoder to perform my task.
> >
> > To first explain the project idea, currently I am working mainly with
> > surveillance footages. Using a traditional video decoder, I convert the
> > video into series of image frames and perform clustering to create frame
> > groups. Then using some performance metric, my algorithm selects a
> > representative frame from each group. By implementing a custom encoder
> and
> > decoder, I aim to improve the speed of extracting these representative
> > frames.
> >
> > To further explain, I would like the custom encoder to be able to force
> the
> > frames I have chosen (representative frames) to be i-frames. The
> assumption
> > is that when analyzing videos, we only need to evaluate these
> > representative frames instead of the whole video (this I have already
> > experimentally confirmed), and extracting i-frames is faster than
> > extracting a combination of i-frames, b-frames, and p-frames.
> >
> > Also, I would like to implement a decoder to quickly extract the series
> of
> > frames I give as input (they will all be i-frames). Through preliminary
> > experiments, I realized that extracting only i-frames by using the
> command
> > 'select pict_type\I' query command only gives a 2x speed boost compared
> to
> > extracting all frames within the video. Hopefully, by implementing a
> custom
> > decoder, I aim to further increase this extraction speed.
> >
> > Is there any code I can refer to? Is my assumption of 'if I were to
> > implement a custom decoder, the extraction speed would be faster than
> using
> > the keyword select to get all the i frames' valid?
> >
> > Any help would be greatly appreciated. Thank you and have a wonderful
> day.
> >
>
> Sorry, but with all due respect, the need for custom decoder does not
> make any sense.
>
> > Sincerely,
> > Jaeho Bang
> >
> > --
> > *Jaeho Bang*
> > *PhD Student in Computer Science*
> > Georgia Institute of Technology
> > North Ave NW, Atlanta, GA 30332, USA
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".



-- 
*Jaeho Bang*
*PhD Student in Robotics*
Georgia Institute of Technology
North Ave NW, Atlanta, GA 30332, USA