[Libav-user] Is a custom I/O read callback allowed to overwrite the buffer if it returns AVERROR_EOF?

[Jyrki Vesterinen]

Hello,
I am developing a game with the Unity game engine. For various reasons, our 
game uses custom asset archives instead of Unity's native assets, and 
utilizes FFmpeg for decoding images and audio inside these custom archives.

To allow FFmpeg to read data from these encrypted asset archives, we give 
it custom I/O read callbacks implemented in C# and made accessible to 
native code via reverse P/Invoke. The C# delegate for the read callback is 
as follows:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate int ReadPacket(IntPtr opaque, [Out, 
MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)] byte[] buf, int 
bufSize);

The "Out" is a directional attribute informing the C# compiler that the 
caller will use contents of the array but the callee doesn't care about its 
initial content. For more information, see 
https://docs.microsoft.com/en-us/previous-versions/dotnet/netframework-4.0/77e6taeh(v=vs.100)?redirectedfrom=MSDN


This means that FFmpeg will always see what my read callback has explicitly 
written to the buffer, but whatever was in the buffer before FFmpeg called 
the callback is not guaranteed to be preserved, even if the read operation 
didn't fill the whole buffer or even read any bytes at all.

I have discovered that one of our Ogg Vorbis music tracks fails to play 
correctly due to this. Logging FFmpeg's calls to the callbacks shows this 
(note: I have given FFmpeg a 1MB buffer):

1. ReadPacket, 1048576 bytes. The length of the entire track is 1029379 
bytes, so we write it into the buffer and return 1029379.
2. Seek, with "whence" set to AVSEEK_SIZE. We return 1029379.
3. ReadPacket, 1048576 bytes. Since we're already at the end of the file, 
we return AVERROR_EOF and P/Invoke overwrites the entire buffer.

After this, oggdec.c:ogg_read_page() fails to find a sync word and FFmpeg 
returns AVERROR_INVALIDDATA.

I can easily fix this by adding the "In" directional attribute, i.e. 
turning the C# delegate into

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate int ReadPacket(IntPtr opaque, [In, Out, 
MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)] byte[] buf, int 
bufSize);

This instructs P/Invoke to preserve the original content of the array when 
calling the function, and thus it remains after the call too. However, it 
comes with a performance penalty.

My question is: is the read callback supposed to preserve content of the 
array, or is it a bug that the Ogg demuxer assumes it to have remained 
after calling the callback if it returns AVERROR_EOF?

-- 
Jyrki Vesterinen
Programmer
NekoNyan Ltd