[FFmpeg-devel] mutlithreading in ffmpeg?

[Thomas Worth]

On Sun, Mar 27, 2011 at 3:08 AM, Reimar Döffinger
<Reimar.Doeffinger at gmx.de> wrote:
> On Sat, Mar 19, 2011 at 09:21:23AM -0700, Thomas Worth wrote:
>> On Sat, Mar 19, 2011 at 5:08 AM, Reimar Döffinger
>> <Reimar.Doeffinger at gmx.de> wrote:
>> > On Fri, Mar 18, 2011 at 05:33:04PM -0700, Thomas Worth wrote:
>> >> On Fri, Mar 18, 2011 at 7:31 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
>> >> > On Thu, Mar 17, 2011 at 04:10:58PM -0700, Thomas Worth wrote:
>> >> >> >> There is ffmpeg-mt for frame level multithreaded decoding.
>> >> >> >> http://gitorious.org/ffmpeg/ffmpeg-mt
>> >> >> >> It would be nice to see that integrated into mainline ffmpeg.
>> >> >> >> Its quite effective, scales nicely to whatever number of cores you've got
>> >> >> >> and not sensitive to what to tool was used to create the content.
>> >> >> >
>> >> >> > It will be integrated soon.
>> >> >>
>> >> >> Hopefully you guys can fix the seeking problem with the H.264 decoder
>> >> >> before then... hint, hint ;-)
>> >> >
>> >> > which issue number on the bug tracker is that?
>> >>
>> >> I didn't file a bug report in case I was overlooking something.
>> >> However, I've spent some time with this and I am pretty sure it's a
>> >> problem with multi-threaded decoding. Anyway, I put together a simple
>> >> page that explains it in case it's a problem with my code:
>> >>
>> >> http://rarevision.com/ffmpeg-mt/
>> >
>> > Well, it is in sofar a problem with your code as it does not show any _bug_,
>> > just some conspiciously inconsistent behaviour.
>> > And even that is unsure since you do not show the full code, e.g. whether
>> > you correctly flush after a seek.
>> > But either way not immediately returning a decoded frame is _not_ a bug,
>> > if your code in any way assumes this it is _broken_ and it will not
>> > work correctly with some current and future (also hw-accelerated) codecs.
>>
>> I have assumed this entire time that the problem was with my code, but
>> after trying several variations to no avail and the fact that it works
>> on regular ffmpeg and not ffmpeg-mt should be the first clue that
>> something is wrong. Besides, why would my code influence things like
>> whether or not the frame immediately after a keyframe (consistently,
>> as shown by the output) gets returned?
>
> I am going to say it again: that in itself is not necessarily a bug.
> The difference to single-thread does make it suspicious though, but nothing
> more.

Of course it's suspicious -- it isn't producing expected results. And
don't confuse "single thread" with simply setting codec context's
thread count to 1. This does not work regardless of whether the thread
count is 1 or > 1. It ONLY works with normal (non-mt) ffmpeg.

> And the fact that here:
>> seekpos:01001|pts:01001|decode2 return val:0319|got_pic_ptr:000|is keyframe:0
>> seekpos:01001|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
> You are seeking to the same position twice and get a different result
> means there's some state lingering. Whether that's your code's fault or
> not is impossible to say with such an incomplete code-snippet.

The only reason this happens is because I am forcing a decode by
ignoring *got_picture_ptr to expose the problem. This would not happen
during normal operation. In fact, with the H.264 decoder I don't even
have to check for got_picture_ptr, because everything returned from
avcodec_decode_video2 is a valid frame. That is, except for ffmpeg-mt,
where each frame after a keyframe is inexplicably ignored EVEN THOUGH
it is reporting the number of bytes in the decode result!

Regarding the incomplete code-snippet, everything else would be set up
normally using typical code for opening and reading using normal
avcodec structs. I didn't include it because I'm not doing anything
special.

>> And why is it returning the
>> same frame twice, even though the timestamps are one frame apart?
>
> None of the information in that page indicates that the same frame is returned.

You did not look closely enough:

seekpos:01001|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
seekpos:02002|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0

>> This
>> works fine on all frames _after_ the one following the keyframe. And
>> why is avcodec_decode_video2() returning the decompressed size of the
>> frame but not actually putting it into an AVFrame?
>
> It returns the amount of data consumed. If it's a B-frame it might be the
> most sensible thing to just discard it.

There are no b-frames in the video (I know I did not make this clear
in the description, but now you know). If it returns the amount of
data consumed, then why not set got_picture_ptr? It does it for every
other frame, just not the one after a keyframe!

>> Anyway, this problem is very easy to reproduce and 100% consistent if
>> anyone is interested in having a look.
>
> It is not, there's no complete, actually compilable code there, nor is
> there anything that indicates what _exactly_ is the bug.
> You're asking us to waste our time doing something you have already done
> just to reproduce a bug that you don't even say what exactly the bug is
> (or at least the thing you say is not a bug in itself). I'd complain about
> that even if I were paid to do it.

I am trying to give the benefit of the doubt to the developers. I
don't know if this is a bug. It seems like one, but I am trying to be
respectful and bring up the issue first before placing blame on the
other party (interesting concept, eh?). If I had to say this was a
bug, then I'd say the bug is related to seeking which is causing
decoding errors -- perhaps there is some internal buffering taking
place upon normal decode loop execution that does not occur when
seeking / specifying timestamps directly. Frankly, I don't know
avcodec code well enough to say one way or another, hence my
reluctance to call this a bug.

And please understand that I'm not _asking_ anybody to do anything. I
am just trying to solve this problem, and maybe someone is interesting
in helping me. If you're not interested and would prefer to not "waste
you time," then by all means, don't. But please, in the mean time
don't tell me my arguments are invalid when you haven't even read them
completely. That does not seem very productive, in my opinion.