[FFmpeg-devel] Custom Video Encoding and Decoding

[Zhao Zhili]

On July 29, 2020 3:13:34 PM GMT+08:00, Jaeho Bang <jaeho.bang at gmail.com> wrote:
>Dear Paul,
>Thank you for the reply! I must have done a bad job explaining, sorry
>about
>that. There is some underlying information that I have left out. My
>initial
>clustering algorithm / representative frame selection algorithm
>actually
>spits out a frame importance hierarchy. I want to implement a custom
>decoder because it needs to process inputs such as 'select 100 most
>important frames of this video'. Rather than decoding the entire video
>and
>choosing the 100 most important frames based on frame importance
>hierarchy,
>I want to only decode the frames that I specify (By having a custom
>encoder, I have already forced these frames to be i-frames, hence
>minimum
>computation is required). The goal of this decoder is to minimize
>memory
>footprint (since decoding entire video into image frames increases
>memory
>usage by 100x according to my experiments) while maximizing decoding
>speed.
>Does this provide a clearer reason why I am thinking of implementing a
>custom decoder? Please feel free to ask for clarifications.
>

You can add custom info to container or out-of-band to help select which packet to decoding. The custom info doesn't belong to the codec normally.

>Thank you,
>Jaeho Bang
>
>On Wed, Jul 29, 2020 at 3:31 PM Paul B Mahol <onemda at gmail.com> wrote:
>
>> On 7/29/20, Jaeho Bang <jaeho.bang at gmail.com> wrote:
>> > Dear members of ffmpeg-devel,
>> >
>> > I am currently working on a research project and from looking over
>the
>> > examples and documentation in the ffmpeg repository, I believe I
>need to
>> > implement a custom encoder / decoder to perform my task.
>> >
>> > To first explain the project idea, currently I am working mainly
>with
>> > surveillance footages. Using a traditional video decoder, I convert
>the
>> > video into series of image frames and perform clustering to create
>frame
>> > groups. Then using some performance metric, my algorithm selects a
>> > representative frame from each group. By implementing a custom
>encoder
>> and
>> > decoder, I aim to improve the speed of extracting these
>representative
>> > frames.
>> >
>> > To further explain, I would like the custom encoder to be able to
>force
>> the
>> > frames I have chosen (representative frames) to be i-frames. The
>> assumption
>> > is that when analyzing videos, we only need to evaluate these
>> > representative frames instead of the whole video (this I have
>already
>> > experimentally confirmed), and extracting i-frames is faster than
>> > extracting a combination of i-frames, b-frames, and p-frames.
>> >
>> > Also, I would like to implement a decoder to quickly extract the
>series
>> of
>> > frames I give as input (they will all be i-frames). Through
>preliminary
>> > experiments, I realized that extracting only i-frames by using the
>> command
>> > 'select pict_type\I' query command only gives a 2x speed boost
>compared
>> to
>> > extracting all frames within the video. Hopefully, by implementing
>a
>> custom
>> > decoder, I aim to further increase this extraction speed.
>> >
>> > Is there any code I can refer to? Is my assumption of 'if I were to
>> > implement a custom decoder, the extraction speed would be faster
>than
>> using
>> > the keyword select to get all the i frames' valid?
>> >
>> > Any help would be greatly appreciated. Thank you and have a
>wonderful
>> day.
>> >
>>
>> Sorry, but with all due respect, the need for custom decoder does not
>> make any sense.
>>
>> > Sincerely,
>> > Jaeho Bang
>> >
>> > --
>> > *Jaeho Bang*
>> > *PhD Student in Computer Science*
>> > Georgia Institute of Technology
>> > North Ave NW, Atlanta, GA 30332, USA
>> > _______________________________________________
>> > ffmpeg-devel mailing list
>> > ffmpeg-devel at ffmpeg.org
>> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> >
>> > To unsubscribe, visit link above, or email
>> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>
>
>
>-- 
>*Jaeho Bang*
>*PhD Student in Robotics*
>Georgia Institute of Technology
>North Ave NW, Atlanta, GA 30332, USA
>_______________________________________________
>ffmpeg-devel mailing list
>ffmpeg-devel at ffmpeg.org
>https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>To unsubscribe, visit link above, or email
>ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

-- 
使用 K-9 Mail 发送自我的Android设备。