[FFmpeg-user] Question about video compositing

[Alex Speller]

I have a question about video compositing. I’ve included the text of the
question below but I’ve also put it in a gist for easier to read formatting
here: https://gist.github.com/alexspeller/aefdd5a6d7100d28d0bbc4838527f797

I have multiple mp4 video files and I want to composite them into a
single video. Each stream is an mp4 video. They are of different
lengths, and each file also has audio.

The tricky thing is, I want the layout to change depending on how many
streams are currently visible.

As a concrete example, say I have 3 video files:

| File  | Duration | Start | End |
|-------|----------|-------|-----|
| a.mp4 | 30s      | 0s    | 30s |
| b.mp4 | 10s      | 10s   | 20s |
| c.mp4 | 15s      | 15s   | 30s |


So at t=0 seconds, I want the video to look like this:

```
   +-------------------------+
   |                         |
   |                         |
   |                         |
   |                         |
   |          a.mp4          |
   |                         |
   |                         |
   |                         |
   |                         |
   |                         |
   +-------------------------+


```

At t=10s, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         b.mp4                |
|                              |
|                              |
|                              |
|                              |
|                              |
+------------------------------+
```

At t=15s, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         b.mp4                |        |
|                              |        |
|                              | c.mp4  |
|                              |        |
|                              +--------+
|                              |
+------------------------------+
```

And at t=20s until the end, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         c.mp4                |
|                              |
|                              |
|                              |
|                              |
|                              |
+------------------------------+
```

Ideally there would be some animated transitions between the states,
but that's not essential.

I have found two possible approaches that might work, but I'm not sure
what the best one is. The first is using
[filters](https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos)
to acheive the result, but I'm not sure if it will cope well with (a)
the changing layouts and (b) keeping the audio without any artefacts
when the layout changes.

The other approach I thought of would be exporting all frames to
images, building new frames with imagemagick, and then layering the
new frames on top of the audio like in [this blog
post](https://broadcasterproject.wordpress.com/2010/05/18/how-to-layerremix-videos-with-free-command-line-tools/).

Any suggestions on if either of these approaches is better, or any
alternatives? Thanks!