[FFmpeg-user] Windows 10, ffmpeg concat/demux is slow... part 2

[Kevin Duffey]

If Kevin can share two reasonable sized sample files, I can try the
ffmpeg operation in my computer. Using Linux, though.

 And with sufficent memory: From RAM disk to /dev/null.


First of all.. sorry for this again.. I installed Thunderbird.. and it doesnt appear to work well if at all with yahoo mail. Then somehow it sent an email to one person on this thread, but now I am getting non stop error messages. Such a pain in the butt. Gmail works fine. Just cant seem to get yahoo configured with imap.
Anywho...

I am not sure what you are asking..exactly. You want me to upload a couple large files some place for you to download and test? I would assume.. if that is what you are asking.. any files should work yah? The files I work with are 140GB in size.. so I would need a month or so to upload two of those.

I am also not entirely sure about testing the copy process.  I did once try to use the Windows copy command and that too took a long while.  Which if that is the case, when copying two files together (using the + option) and I am doing so on an SSD.. it strikes me as a possible Windows OS Filesystem issue. My main SSD is NVMe.. write speed of 1500MB/s, so no way should it take very long to copy 300GB.. a few minutes tops.. not the hours and hours it currently seems to take. 

Even when I copy over USB 3.0 or 3.1, at 5Gb/s, the bottleneck should be the recording medium. Given that the external USB 3.1 devices are typically using SATA3 interface internally, and with an m.2 ssd in it, it should still handle 250MB/s or more at full SATA3 speeds.. basically the max speed of the m.2 ssd write speed. Naturally this could vary, but my m.2 claims 450MB/s write, 550MB/s read, and SATA 3 at 6gp/s with parity.. is still approaching 600 MB/s theoretical speeds. So even if I allow for 1/2 that for any number of issues.. I would hope I could see 250MB/s or more throughput. Which on 2 x 150GB files, should still yield a copy speed of about 5 minutes or so, tops.. if not less.

So..that has me questioning.. what in the OS maybe I need to change so that copies are much faster.

I just cant imagine if ffmpeg concat copy process is relatively based on the speed of copying files, why I am stuck at 12 or more hours.. when disk medium speed wise, it shouldnt take more than 5 minutes.

    On Monday, November 13, 2017, 2:20:07 PM PST, Carlos E. R. <robin.listas at telefonica.net> wrote:  
 
 -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Monday, 2017-11-13 at 15:56 -0000, Kevin Duffey wrote:

> [quote]
>> As one of the drives
>> is an SSD.. this seems ridiculous to me that it is basically moving at
>> USB2 speeds. At the very least, as it is over a USB 3.1 gen 2, 10Gb/s
>> wire... it should be much much faster than this. So now I am left
>> wondering why my system is super slow at copying files. Is there some
>> configuration in Windows 10 that has to be enabled to allow fast
>> copying?
>
> I don't know. In the case of the external disk above it seems a hardware
> limitation. Either the disk or the USB3 interface of the disk or the
> computer.
> [/quote]
>
> ok.. see if the quote thing worked lol.

Perfectly :-)

> So just to be clear, I have been building computers, networks, etc for 
> many years now.. only saying that to ensure that for the most part, I 
> understand the various speeds that USB, SATA, etc should be able to do. 
> Not that anyone was totally questioning that of me, but wanted to throw 
> out there that I do have the knowledge of the basic hardware stuff.  It 
> is in particular with ffmpeg that I am blundering.

But if I remember correctly this computer also has problems with a simple 
concat on the command line:

copy file1.mov + file2.mov fileout.mov

Do this on the same directory, to test a single disk. You can repeat on 
another disk, to try.


> With that in mind, lets assume I had nothing but SSD drives. My laptop 
> has 2 NVMe 960 EVO drives, and one SATA3 SSD. For clarity, it is a SAGER 
> desktop replacement laptop.. not very portable. Also has a 2TB Firecuda 
> SSHD and 64GB RAM with 6700K cpu.

It should be very fast, speciallty the NVMe's. Are they configured in a 
RAID? Is it perhaps degraded?

The SATA3 SSD should also be very fast.

> What sort of speeds should I see when concatting two DNxHR SQ 4K videos 
> together?  That would help me understand better what to expect. I would 
> assume from some other posts I found around the interwebs, that I should 
> see 150fps to 300fps with this process, not what i see now at a paltry 
> 9fps.  If this is a CPU bound task.. the CPU indicates it is using 1% 
> total.. but the disk usage shows at about 20%. So I would assume that I 
> am still not being hampered by disk i/o given that it is not maxed out.

If you can somehow post the source videos somewhere, or other videos with 
the exact same encoding, I can try. In Linux, which is what I use, and 
with more modest hardware.

> I will be building my threadripper system in a couple weeks.. I hope 
> that ffmpeg will benefit from the 16 cores and be much faster... 
> assuming I can get past whatever the issue is with why it is so slow 
> now.

Well, you need finding what the issue is.

- -- 
Cheers,
        Carlos E. R.
        (from openSUSE 42.2 x86_64 "Malachite" at Telcontar)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAloKGoUACgkQtTMYHG2NR9WCuACcDulNAxF65ynWzF0TwKNwOeJ1
71QAnjPfqx9tYgwj3AKcEakAc33BslWU
=U4X1
-----END PGP SIGNATURE-----
_______________________________________________
ffmpeg-user mailing list
ffmpeg-user at ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".