[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

Emanuele Oriani ema at fastwebnet.it
Tue May 19 13:25:12 EEST 2020


Hi Lynne,

Thanks for the feedback.
Some more discussion points below.

 > There already is a zero-overhead capture on linux - kmsgrab. It works 
on AMD and Intel.

Does it work on Nvidia too? Does it have smooth capture?
Does it work for 3d applications?

 >> Wouldn't a hwaccel frame imply no choice for encoder (as in now we 
get a true uncompressed RGBA buffer which can be encoded as we see fit - 
if we were to get hwacell wouldn't we be forced to use the encoded data 
as is)?
 >>
 >
 > No.

My point was around if you use say NVenc to compress frames, you 
wouldn't want to decompress-recompress to avoid artefacts?

 > I wouldn't be surprised if the xlib code has some PTS issues and 
schedules downloading a frame late.

I'm far from being an expert, but I would be surprised if that had 
issues considering it's the main Linux screen capture we have?
The xcb calls to grab a frame are quite simple/straightforward (both shm 
and non shm versions).

 > Maybe its worth fixing that instead of adding a yet another x11 capture?

This is a xcomposite capture, I would say it's different than a x11 raw 
capture.

 > I'm against any OpenGL code being ran anywhere in our libraries since 
it will disturb OpenGL's global
 > state, breaking any OpenGL users of our libraries (there are quite a 
lot). Yes, there are also people who
 > want to screencapture and use OpenGL. I'm one.
 > This is also why we don't have OpenGL filtering in libavfilter.
 > So that's a big NAK for me.

Fair enough - Understood why it would be a NAK from you.
Would there be a way to mark this as experimental?

Out of curiosity if I was proposing/using the native NVidia proprietary 
library for screen capture (linked at runtime dynamically), would that 
be more acceptable in terms of conflicts (I wouldn't like it because it 
wouldn't support AMD/Intel based hardware)?

 > As-is now, I don't see much value in this patch.

I wrote this because I wanted to capture 3d apps at 60 FPS with high 
quality without having to drop frames.
If there is a better way with ffmpeg/libav*, please feel free to suggest 
- otherwise currently folks would be forced to use OBS and not use 
ffmpeg/libav* based solutions.

The video games segment under Linux is expanding, would ffmpeg/libav* 
have the right infra/codecs to cater for it?
Can we capture true 60 FPS of a 3d app today? Do we have native 
Vulkan/OpenGL capturers?

If yes then I would agree with you (no point in having this patch), 
otherwise there is value in it.

On 19/05/2020 09:33, Lynne wrote:
> May 19, 2020, 09:23 by ema at fastwebnet.it:
> 
>> Hi Marton,
>>
>> Thanks very much for the feedback; below answers to your points - let me know further feedback if any.
>>
>>> And sorry, I cannot say how useful this would be, maybe now is the time
>>> for people to speak up if somebody is particularly against adding this
>>> for any reason.
>>>
>>
>> I haven't been able to capture video-games/3d apps running in full screen with x11grab, and when running in windowed mode the capture was sub optimal in terms of quality (lost frames/choppy/etc etc).
>> Unless we have better solutions with ffmpeg/libav* (which I'm not aware of :) this xcompgrab would target such audience (smooth capture alas more CPU usage).
>> But again, if there's already a better capture, then this has been an academic exercise :)
>>
> 
> There already is a zero-overhead capture on linux - kmsgrab. It works on AMD and Intel.
> 
> 
> 
>>> - Is there a way to keep the captured textures as an
>>> OpenGL/VAAPI/NVenc/Vulkan object, so the frames can work as some kind of
>>> hwaccel frames? Can this improve performance?
>>>
>>
>> I think there would be. I would need to find more in terms of documentation for both hwaccel frames structures/management.
>> Please feel free to point me to guides/code/samples etc etc.
>>
> 
> Uploading to hardware frames is something uses can do themselves already. Unless there is a native
> interop, I don't see how doing the upload in the avdevice helps.
> 
> 
> 
>> Wouldn't a hwaccel frame imply no choice for encoder (as in now we get a true uncompressed RGBA buffer which can be encoded as we see fit - if we were to get hwacell wouldn't we be forced to use the encoded data as is)?
>>
> 
> No.
> 
> 
> 
>>> - What can be the reason between the quality/smoothness difference of
>>> x11grab and the Compositor capturer?
>>>
>>
>> My suspicion is that OpenGL has access to the buffered data on the videocard itself, hence able to get smooth frames.
>>
>> Related to both these questions, I've asked the same on Stackoverflow:
>>
>> https://stackoverflow.com/questions/61613441/is-there-a-better-more-efficient-way-to-capture-composite-x-windows-in-linux
>>
>> Unfortunately no one has been able to give me an answer (I did set a bounty for it) so I posted my own understanding.
>>
> 
> I wouldn't be surprised if the xlib code has some PTS issues and schedules downloading a frame late.
> Maybe its worth fixing that instead of adding a yet another x11 capture?
> 
> 
> 
>>> - Maybe some openGL glue can be shared with libavdevice/opengl_enc.c?
>>> Take a look, I am not sure, because that was implemented with SDL and
>>> cross platform capabilities in mind, but since your capture device is
>>> only for linux (or not?), then maybe it makes more sense to keep it
>>> separete?
>>>
>>
>> This is indeed a very specific implementation for Linux, I would agree that perhaps not sure this makes sense to use shared OpenGL functions?
>>
> 
> I'm against any OpenGL code being ran anywhere in our libraries since it will disturb OpenGL's global
> state, breaking any OpenGL users of our libraries (there are quite a lot). Yes, there are also people who
> want to screencapture and use OpenGL. I'm one.
> This is also why we don't have OpenGL filtering in libavfilter.
> So that's a big NAK for me.
> 
> As-is now, I don't see much value in this patch.
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 


More information about the ffmpeg-devel mailing list