[FFmpeg-user] Invalid NAL unit size, S3 Invalid data found when processing input

[Christoph Burgmair]

Hi,
yesterday at 11:20 CET we discovered problems with our AWS batch transcoding.
The EC2 machines that execute the transcoding tasks have problems accessing the mp4 source files by streaming directly from S3.
These streaming / access problems result in FFmpeg transcoding problems.
This is the FFmpeg command we used:

ffmpeg -y -i "$(aws s3 presign s3://<bucket-name-hidden>/VX-926874.mp4)" -an -codec:v libx264 -profile:v main -preset veryslow -sws_flags lanczos -s:v 640x360 -aspect 16:9 -b:v 500k -maxrate:v 500k -bufsize:v 500k -keyint_min 50 -g 50 -x264opts nal-hrd=cbr:pic-struct:no-scenecut:colorprim=bt709:transfer=bt709:colormatrix=bt709:open_gop=0:ref=4 -pix_fmt yuv420p -an -async 1 -metadata:s language=zxx -movflags faststart -loglevel verbose /out/aedaa15d-1b6b-4178-ab5d-d6d845dab205_2019-08-2609:25:10.922542_868803588-402356695-74297555.mp4

Here are the resulting logs:

...
frame=26164 fps= 77 q=27.0 size= 63488kB time=00:17:23.64 bitrate= 498.3kbits/s speed=3.07x 

frame=26188 fps= 76 q=27.0 size= 63744kB time=00:17:24.60 bitrate= 499.9kbits/s speed=3.02x 

[tls @ 0x1c10fc0] error:00000000:lib(0):func(0):reason(0)

[h264 @ 0x1e58840] Invalid NAL unit size (63816 > 51433).

[h264 @ 0x1e58840] Error splitting the input into NAL units.

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1c05380] stream 0, offset 0xaaa1ca47: partial file

https://<bucket-name-hidden>.s3.amazonaws.com/VX-926874.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Date=20190826T092511Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=<tocken-hidden>: Invalid data found when processing input

frame=26208 fps= 40 q=26.0 size= 63744kB time=00:17:25.40 bitrate= 499.5kbits/s speed=1.61x 

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1c05380] stream 0, offset 0xaaa375a4: partial file

https://<bucket-name-hidden>.s3.amazonaws.com/VX-926874.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Date=20190826T092511Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=F<tocken-hidden>: Invalid data found when processing input

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1c05380] stream 0, offset 0xaaa46722: partial file
…


First, the transcoding runs quite smoothly, but after a while we geht the error "Invalid NAL unit size (63816 > 51433).“ as seen above.
The strange thing is that this error only happens very rarely. Usually everything is fine and the files are fully transcoded. But sometimes we get these errors which result in no fully transcoded output files.
From AWS side everything was ok.

I hope someone can help me here.

Best regards,
Christoph