[FFmpeg-devel] [PATCH]lavu/hwaccel_vaapi: Add a fixme for the missing byte_order check

[Mark Thompson]

On 06/10/16 09:05, Carl Eugen Hoyos wrote:
> 2016-10-05 21:55 GMT+02:00 Mark Thompson <sw at jkqxz.net>:
> 
> What I meant is:
> How can "LE" make sense for an 8bit planar format?

It does not.  So, another reason to ignore that field.

> (And what does it tell us about the author?)
> This of course assumes that "YV12" is planar, if it
> isn't, I simply misunderstand the whole code.

Yes, YV12 is planar: it is ffmpeg YUV420P with the two chroma planes swapped.

>> For example, I can do screen capture in X with:
>>
>> ffmpeg -y -vaapi_device /dev/dri/renderD128 -video_size 1920x1080 -framerate 30 -f x11grab -i :0 -vf 'hwupload,scale_vaapi=1920:1080:nv12' -c:v h264_vaapi out.mp4
>>
>> which captures in bgr0, uploads it to the GPU, converts it to
>> nv12 and encodes it as H.264 there.  Alternatively, I can add
>> 'format=rgb0' at the start of the filter chain to convert and
>> upload in a different RGB format, and that produces the
>> correct output too.
> 
> Thank you for confirming this.
> 
> Do you think vaapi's P010 should be mapped to FFmpeg's
> P010LE instead of P010?

P010 is defined as format with a 16-bit unit size, so the native format on a BE
system should be P010BE.  I admit that confusion is likely, though, given that
the actual hardware may be a graphics card which normally works with an LE host.
 We may need to look somewhere else for the answer at that point (possibly the
byte_order field, assuming the drivers manage to fill it correctly in such cases).

I would prefer to only consider this problem once we have some working system to
test on.  On the other hand, if you wish to submit a patch changing it now I
would not mind - it would have no effect on current use because the one system
with working 10-bit support is LE-only.

Thanks,

- Mark