[FFmpeg-devel] mutlithreading in ffmpeg?

[Thomas Worth]

On Tue, Mar 29, 2011 at 6:34 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
> On Mon, Mar 28, 2011 at 07:29:30PM -0700, Thomas Worth wrote:
>> On Sun, Mar 27, 2011 at 3:08 AM, Reimar Döffinger
>> <Reimar.Doeffinger at gmx.de> wrote:
>> > On Sat, Mar 19, 2011 at 09:21:23AM -0700, Thomas Worth wrote:
>> >> On Sat, Mar 19, 2011 at 5:08 AM, Reimar Döffinger
>> >> <Reimar.Doeffinger at gmx.de> wrote:
>> >> > On Fri, Mar 18, 2011 at 05:33:04PM -0700, Thomas Worth wrote:
>> >> >> On Fri, Mar 18, 2011 at 7:31 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
>> >> >> > On Thu, Mar 17, 2011 at 04:10:58PM -0700, Thomas Worth wrote:
>> >> >> >> >> There is ffmpeg-mt for frame level multithreaded decoding.
>> >> >> >> >> http://gitorious.org/ffmpeg/ffmpeg-mt
>> >> >> >> >> It would be nice to see that integrated into mainline ffmpeg.
>> >> >> >> >> Its quite effective, scales nicely to whatever number of cores you've got
>> >> >> >> >> and not sensitive to what to tool was used to create the content.
>> >> >> >> >
>> >> >> >> > It will be integrated soon.
>> >> >> >>
>> >> >> >> Hopefully you guys can fix the seeking problem with the H.264 decoder
>> >> >> >> before then... hint, hint ;-)
>> >> >> >
>> >> >> > which issue number on the bug tracker is that?
>> >> >>
>> >> >> I didn't file a bug report in case I was overlooking something.
>> >> >> However, I've spent some time with this and I am pretty sure it's a
>> >> >> problem with multi-threaded decoding. Anyway, I put together a simple
>> >> >> page that explains it in case it's a problem with my code:
>> >> >>
>> >> >> http://rarevision.com/ffmpeg-mt/
>> >> >
>> >> > Well, it is in sofar a problem with your code as it does not show any _bug_,
>> >> > just some conspiciously inconsistent behaviour.
>> >> > And even that is unsure since you do not show the full code, e.g. whether
>> >> > you correctly flush after a seek.
>> >> > But either way not immediately returning a decoded frame is _not_ a bug,
>> >> > if your code in any way assumes this it is _broken_ and it will not
>> >> > work correctly with some current and future (also hw-accelerated) codecs.
>> >>
>> >> I have assumed this entire time that the problem was with my code, but
>> >> after trying several variations to no avail and the fact that it works
>> >> on regular ffmpeg and not ffmpeg-mt should be the first clue that
>> >> something is wrong. Besides, why would my code influence things like
>> >> whether or not the frame immediately after a keyframe (consistently,
>> >> as shown by the output) gets returned?
>> >
>> > I am going to say it again: that in itself is not necessarily a bug.
>> > The difference to single-thread does make it suspicious though, but nothing
>> > more.
>>
>> Of course it's suspicious -- it isn't producing expected results. And
>> don't confuse "single thread" with simply setting codec context's
>> thread count to 1. This does not work regardless of whether the thread
>> count is 1 or > 1. It ONLY works with normal (non-mt) ffmpeg.
>>
>> > And the fact that here:
>> >> seekpos:01001|pts:01001|decode2 return val:0319|got_pic_ptr:000|is keyframe:0
>> >> seekpos:01001|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
>> > You are seeking to the same position twice and get a different result
>> > means there's some state lingering. Whether that's your code's fault or
>> > not is impossible to say with such an incomplete code-snippet.
>>
>> The only reason this happens is because I am forcing a decode by
>> ignoring *got_picture_ptr to expose the problem. This would not happen
>> during normal operation. In fact, with the H.264 decoder I don't even
>> have to check for got_picture_ptr, because everything returned from
>> avcodec_decode_video2 is a valid frame. That is, except for ffmpeg-mt,
>> where each frame after a keyframe is inexplicably ignored EVEN THOUGH
>> it is reporting the number of bytes in the decode result!
>
> you have to check got_picture_ptr and always had to check it.
> the data you get if got_picture_ptr is 0 is undefined
>
> and if i understand you correctly (please correct me if i misunderstand)
> you talk about this undefined data being inconsistent.

Yes, I normally check got_picture_ptr. However, I bypass this on
purpose to force the return of data when I pass an arbitrary
timestamp. This doesn't seem to be a problem with normal ffmpeg
(usually get_picture_ptr is set on every timestamp). In the case of
ffmpeg-mt, got_picture_ptr is 0 following a keyframe, but
avcodec_decode_video2 is still returning the number of bytes processed
and it is not the same as the previous frame. So, it is not that a
previous value is being buffered. I think it is actually processing
the frame (as normal ffmpeg is), but neither setting got_picture_ptr
nor filling the AVFrame.

Also, I get the following error after seeking directly to a frame
following a keyframe:

[h264 @ 0x100819400] Missing reference picture
[h264 @ 0x100819400] decode_slice_header error

I get this error, but I STILL get a frame correctly with ffmpeg. With
ffmpeg-mt, it skips it and returns the next frame instead:

seekpos:00000|pts:00000|decode2 return val:4883|got_pic_ptr:328|is keyframe:1
seekpos:01001|pts:01001|decode2 return val:0319|got_pic_ptr:000|is keyframe:0
seekpos:01001|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
seekpos:02002|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0

Notice that on frame 2 (timestamp 1001), the return value from the
decode function is 319, which is different from the next P frame,
which is 318. Now of course we get 318 twice at PTS 2002 because
ffmpeg-mt is returning the same frame twice, even though two different
timestamps are specified.

>> Regarding the incomplete code-snippet, everything else would be set up
>> normally using typical code for opening and reading using normal
>> avcodec structs. I didn't include it because I'm not doing anything
>> special.
>>
>> >> And why is it returning the
>> >> same frame twice, even though the timestamps are one frame apart?
>> >
>> > None of the information in that page indicates that the same frame is returned.
>>
>> You did not look closely enough:
>>
>> seekpos:01001|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
>> seekpos:02002|pts:02002|decode2 return val:0318|got_pic_ptr:328|is keyframe:0
>>
>> >> This
>> >> works fine on all frames _after_ the one following the keyframe. And
>> >> why is avcodec_decode_video2() returning the decompressed size of the
>> >> frame but not actually putting it into an AVFrame?
>> >
>> > It returns the amount of data consumed. If it's a B-frame it might be the
>> > most sensible thing to just discard it.
>>
>> There are no b-frames in the video (I know I did not make this clear
>> in the description, but now you know). If it returns the amount of
>> data consumed, then why not set got_picture_ptr? It does it for every
>> other frame, just not the one after a keyframe!
>
> probably because theres 1 frame delay
> if this changed over the ffmpeg-mt merge then we might have a bug in
> there but 1 frame delay streams (without B frames) are rather common
> practically all mpeg2 without b frames fall in this category
>
> can you confirm that the delay behavior changed?
> if yes i need a reproduceable test case to look into it

Absolutely. I have attached a sample program and makefile, which
should show the phenomenon. If you compile with normal ffmpeg
libraries and then run a test, you will see that the PTS values
returned are consistent with timestamp values passed to av_seek_frame.
If you compile with ffmpeg-mt, you will see that there is a "delay"
(i.e. error) even if you set threads to "1" and even if you compile
without pthreads.

Hopefully the attached code will help both you and Reimar see what the
issue is. I have documented the code somewhat, but if you need more
information I will try to help.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lavtest.zip
Type: application/zip
Size: 3027 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20110329/0c159113/attachment.zip>