[FFmpeg-user] 'mix' filter questions

[Mark Filipak (ffmpeg)]

Synopsis: I present the 'mix' filter (in compact format) followed by criticism followed by my 
rewrite followed by remaining questions.

Tip: Before responding, I suggest you save time and effort by reading below the divider (=====). If 
I have it right, then kindly say so; otherwise, let me know what I have wrong.

If our email clients mangle the message formatting, I apologize. I've attached an unmangled version.

Thanks a lot!
Mark.

PS: I'm asking for help, not because I'm lazy, but because I'm overloaded and you folks are probably 
very familiar with the filter and can probably confirm/refute what I write from the top of your 
heads without taking much time. -- M.


mix   ...Mix several video input streams into one video stream.
mix=nb_inputs=2:weights=Ø:scale=Ø:duration=longest   ...Defaults (Ø means no default).
               :         :       :          :
               :         :       :          Specify how end of stream is determined.
               :         :       :          longest    ...The duration of the longest input.
               :         :       :          shortest   ...The duration of the shortest input.
               :         :       :          first      ...The duration of the first input.
               :         :       :
               :         :       Specify scale, if it is set it will be multiplied with sum of each 
weight multiplied with pixel values to give final destination pixel value. By default scale is auto 
scaled to sum of weights.
               :         :
               :         Specify weight of each input video stream as sequence. Each weight is 
separated by space. If number of weights is smaller than number of frames last specified weight will 
be used for all remaining unset weights.
               :
               The number of inputs.

Criticism (re, "Specify scale"):
   "if it is set[,] it will be multiplied with sum of each weight multiplied with pixel values to 
give final destination pixel value"
   That vaguely describes how the filter uses weights but not what scale is.
   The more I read that sentence, and the more I try to form a mental picture of the process, the 
less sense it makes.
Criticism (re, scale):
   Is there one scale for all inputs or one scale per input?
Criticism (re, "Specify scale"):
   "sum of each weight" -- Really? What does "sum" of "each" mean?
   Assuming that what's intended is "sum of [all/the] weights", why would you multiply pixel values 
by the sum of weights? -- all pixel values would be multiplied by the same total weight. That 
wouldn't make sense, would it?
Criticism (re, "Specify scale"): "By default[,] scale is auto scaled to sum of weights." "scale is 
auto scaled" -- Really? What does that really mean?
Criticism (re, "Specify weight"): "If number of weights is smaller than number of frames" -- Huh? 
"frames"? Is there one weight per frame? ...That would make for a truly gigantic command line, eh?

=====

mix   ...Combine multiple streams via weighted, scaled, pixel mixing.
Changes frame rate? No, Frames? No, Fields? No, Pixels? Yes.

Process: for each input: '[A]' '[B]' etc.,
           for each frame,
            for each pixel: 'pixel_A' 'pixel_B' etc.,
             input pixels --> [A] pixel_A x weight_A x scale_A --------.
             input pixels --> [B] pixel_B x weight_B x scale_B ---.    ¦
                               :                                  ¦    ¦
             input pixels --> [Z] pixel_Z x weight_Z x scale_Z --(+)--(+)--> output pixel

               The number of inputs.
               :          first      ...End the output at the end of the first input.
               :          shortest   ...End the output at the end of the shortest input.
               :          longest    ...End the output at the end of the longest input.
               :          :
mix=nb_inputs=2:duration=longest:weights=?:scale=[note 1]   ...Defaults (? means no default).
                                          :       :
                                          :       Space-delimited number list [note 1].
                                          Space-delimited number list [note 2].

[note 1] One scale per input. If 'scale' is missing, a per-input (default) scale is applied: scale = 
weight/(sum of all assigned weights).
[note 2] One weight per input. At least one weight must be listed, but not all inputs require that a 
weight be listed. If there are more inputs than weights, the last weight in the list is assigned to 
the remaining inputs.

Remaining questions:
What is 'scale' datatype?
What is 'weight' datatype?
Can inputs have differing frame rates? Differing frame sizes? Differing pixel formats?
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mix.filter.txt
URL: <https://ffmpeg.org/pipermail/ffmpeg-user/attachments/20210319/6b827e94/attachment.txt>