[FFmpeg-user] Seeking smart decomb filter

[Mark Filipak (ffmpeg)]

On 01/21/2021 09:04 AM, Paul B Mahol wrote:
> On Thu, Jan 21, 2021 at 2:34 PM Mark Filipak (ffmpeg) <markfilipak at bog.us>
> wrote:
> 
>> On 01/21/2021 07:54 AM, Paul B Mahol wrote:
>>> On Thu, Jan 21, 2021 at 1:51 PM Mark Filipak (ffmpeg) <
>> markfilipak at bog.us>
>>> wrote:
>>>
>>>> On 01/21/2021 07:31 AM, Paul B Mahol wrote:
>>>>> On Thu, Jan 21, 2021 at 12:51 PM Mark Filipak (ffmpeg) <
>>>> markfilipak at bog.us>
>>>>> wrote:
>>>>>
>>>>>> On 01/20/2021 04:50 PM, Paul B Mahol wrote:
>>>>>>> On Wed, Jan 20, 2021 at 10:45 PM Mark Filipak (ffmpeg) <
>>>>>> markfilipak at bog.us>
>>>>>>> wrote:
>>>>>>>> On 01/20/2021 06:17 AM, Paul B Mahol wrote:
>>>>>>>>> Motion compensation does not work that way.
>>>>>>>>
>>>>>>>> Thank you, Paul. Yes, I knew that. I'm not seeking motion
>>>> compensation.
>>>>>>>> Kindly reread my filter
>>>>>>>> features and suggest what comes closest. What I seek simply works on
>>>>>>>> pixels. I'm relying on your
>>>>>>>> experience (and perhaps the experiences of others here).
>>>>>>>
>>>>>>> Have you tried nnedi or estdif filters? Both are intra only
>>>>>> deinterlacers.
>>>>>>
>>>>>> Thank you for the guidance. I'm trying estdif first because it seems
>>>>>> simpler, but it doesn't do the
>>>>>> good thing. It does decomb very effectively, but it does it by
>> aligning
>>>>>> (shifting) the edges in
>>>>>> field'2' to the edges in field'1' (which are not shifted). Aligning
>> with
>>>>>> field'1' produces judder
>>>>>> (or in my case, doesn't eliminate judder). What I want it to do is
>> align
>>>>>> both field'1' edges and
>>>>>> field'2' edges to the median (i.e. half way between the input's
>> edges).
>>>> I
>>>>>> do realize that such a
>>>>>> method would introduce combing along the edges of the frame, but
>> that's
>>>>>> okay.
>>>>>>
>>>>>> Here's the command I used to transcode 24FPS to 60FPS:
>>>>>> ffmpeg -i 24[1][2].mkv -map 0 -filter_complex "telecine=pattern=5,
>>>>>> split[1][2],
>>>>>> [1]select=not(eq(mod(n\,5)\,2))[3], [2]select=eq(mod(n\,5)\,2),
>>>>>> estdif=mode=frame[4],
>>>>>> [3][4]interleave" -codec:v libx265 -x265-params "crf=16:qcomp=0.60"
>>>>>> -codec:a copy -codec:s copy
>>>>>> 24[1][2]-to-60[1][1][estdif=mode=frame(1~2)][2][2].mkv
>>>>>>
>>>>>> I'll try nnedi. Perhaps I can figure it out.
>>>>>>
>>>>>
>>>>> Do not even try.
>>>>> Mentioned filters work only with interlaced frames, while your use case
>>>> is
>>>>> completely invalid and frowned upon.
>>>>> I deeply regret replying to this thread.
>>>>
>>>> I did get nnedi to work but with similar results to estdif.
>>>>
>>>> Kindly educate me: What do you mean by "interlaced frames"? Aren't all
>>>> frames that come out of the
>>>> decoder and into the filter chain interlaced?
>>>>
>>>> Why do you say that my use case is invalid? telecine=pattern=5 works
>> very
>>>> well -- there's no judder
>>>> at all! And if, for the combed frame, I use pp=linblenddeint, it's even
>>>> better, far better than what
>>>> either the NVIDIA GPU or the TV does when fed a 24FPS source and of
>> course
>>>> far, far better than
>>>> telecine=pattern=23.
>>>>
>>>> Please educate me. Why are my stunning results invalid?
>>>>
>>>
>>> Because you are trying to interpolate new frames using telecine filter.
>>
>> I am? I apologize for not being clear...
>>
>> The source is 24FPS. 'telecine=pattern=5' makes a 60FPS target with frames
>> 0 & 1 == source frame 0,
>> and frames 3 & 4 == source frame 1. Target frame 2 (i.e. the target frame
>> in the middle) is combed
>> of course. What I'm trying to do is decomb it. How is that interpolating
>> new frames? Educate me. And
>> if it is interpolating new frames, what's wrong with that? -- It looks
>> stunning on a 60Hz TV whereas
>> 24FPS fed to a 60Hz TV has serious judder.
>>
> 
> You want to decomb telecined frames. But telecine already used existing
> frames to create combed frames.
> So you actually seek frame that is between these two fields in combed
> frame, you cant create it by some nonsense tricks.

I do understand, and I do respect your opinions. May I update?

Using only minterpolate was good, and I saw what it was doing, but it was twice as fuzzy as what I 
already did with the fancy split+select+modulo+interleave that you so dislike. So I tried a 
combination: split before {telecine pattern 5 and minterpolate in parallel} and use the 
select+modulo to pluck out only the middle minterpolate frame, to wit:

ffmpeg -i 24[1][2].mkv -map 0 -filter_complex "split[1][2], [1]telecine=pattern=5, 
select=not(eq(mod(n\,5)\,2))[3], [2]minterpolate=fps=60000/1001:mi_mode=blend, 
select=eq(mod(n\,5)\,0)[4], [3][4]interleave" -codec:v libx265 -x265-params "crf=16:qcomp=0.60" 
-codec:a copy -codec:s copy 24[1][2]-to-60[1][1][minterpolate+select=0(1~2)][2][2].mkv

That worked for 2 or 3 sets of 5 frames, but the minterpolate-split soon fell behind. The longer the 
video, the more that the minterpolate-split fell behind, further and further behind. The sample 
video is only 24 seconds but by the end of it, the telecine-split and the minterpolate-split were so 
out of sync that it was actually painful to watch. I have no idea why the minterpolate-split was 
'running' slow. Do you?

You obviously see what I'm trying to do. Simply pp=linblenddeint'ing the combed frame works pretty 
damn well -- there's only 1 combed frame in a set of 6 frames. And apparently a pp blend is the best 
that can currently be done.

But how difficult would this:

BEFORE
          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

AFTER
             XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

be? If amount of combing could be divided into (global) combing-due-to-panning and (local) 
combing-due-to-object-motion, and the combing-due-to-panning was compensated first, then second, the 
combing-due-to-object-motion applied, I think the result would be fantastic. It would only need to 
be done between line-pairs, and only horizontally (not vertically, as in cubic). Give it a thought, 
Paul. I'd write it if I knew 'C'. If you'd like a commission, I'll go for it. ...Let me know, eh?


>>>> Regards,
>>>> Mark.
>>>>
>>>>>> Thanks again,
>>>>>> Mark.
>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mark.
>>>>>>>>
>>>>>>>>> On Wed, Jan 20, 2021 at 2:45 AM Mark Filipak (ffmpeg) <
>>>>>>>> markfilipak at bog.us>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello All,
>>>>>>>>>>
>>>>>>>>>> I seek a decomb filter that operates on H/2 number of line pairs:
>>>>>> lines
>>>>>>>> i
>>>>>>>>>> & i+1 (where i=0..H/2-1),
>>>>>>>>>> by aligning edges in the two lines at x = delta-x(edge)/2 (i.e.
>> the
>>>>>>>>>> median). The ideal filter would
>>>>>>>>>> differentiate overall motion due to panning versus local motion
>> due
>>>> to
>>>>>>>>>> local object motion.
>>>>>>>>>>
>>>>>>>>>> Pan-combing on the left & right edges is acceptable but it would
>> be
>>>> a
>>>>>>>>>> great bonus if the filter
>>>>>>>>>> performed blend on those edges with the blending radius equal to
>>>>>>>> one-half
>>>>>>>>>> the pan-comb.
>>>>>>>>>>
>>>>>>>>>> Vertical, line-pair-to-line-pair processing is not needed or
>>>> desired.
>>>>>>>>>>
>>>>>>>>>> Frame-to-frame processing is not needed or desired.
>>>>>>>>>>
>>>>>>>>>> I have been trying a great number of filters based on their names
>>>> and
>>>>>>>> what
>>>>>>>>>> is in the documentation.
>>>>>>>>>> The best I've found so far is pp=linblenddeint but it obviously
>>>>>> doesn't
>>>>>>>>>> align edges via the edge
>>>>>>>>>> median and it obviously isn't aware of pan-combing versus
>>>>>>>>>> object-motion-combing.
>>>>>>>>>>
>>>>>>>>>> Any suggestions are very welcome. If I find this filter I will
>> post
>>>> a
>>>>>>>> link
>>>>>>>>>> to a 60FPS telecined
>>>>>>>>>> video that will amaze everyone -- better than anything that Cuda
>> can
>>>>>> do.
>>>>>>>>>>
>>>>>>>>>> -Mark.