[Libav-user] HEVC 2/3 of the frame is green
Michael Wood
mornwood23 at gmail.com
Fri Mar 31 14:52:19 EEST 2023
Hi Srinivas,
Thanks for your reply.
The live555 library is outputting NAL units, which should be sent to the
decoder. I am not sure if i'm supposed to merge multiple nal units with the
same timestamp before sending it to the decoder or is libav smart enough to
figure out that those are fragments and should be decoded together ?
Best,
Milos
On Thu, Mar 30, 2023 at 4:17 PM srinivas gudumasu <srinu.try at gmail.com>
wrote:
> Hi,
>
> Refer https://www.rfc-editor.org/rfc/rfc7798.html clause 4.4.3 (Fragmentation
> Unit).
>
> When you see the PayloadHdr type as 49 it's a Fragmentation Unit RTP
> packet. The FU header has two fields S and E. S bit is set for the first Fragmentation
> Unit of a NAL Unit and E bit is set for the last Fragmentation Unit of a
> NAL Unit.
>
> 0 1 2 3
> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | PayloadHdr (Type=49) | FU header | DONL (cond) |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
> | DONL (cond) | |
> |-+-+-+-+-+-+-+-+ |
> | FU payload |
> | |
> | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | :...OPTIONAL RTP padding |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>
>
> +---------------+
> |0|1|2|3|4|5|6|7|
> +-+-+-+-+-+-+-+-+
> |S|E| FuType |
> +---------------+
>
> Figure 10: The Structure of FU Header
>
> Another way of detecting the last packet of a frame is using a Marker bit. M bit is set to 1, for the last packet of the access unit, carried in the current RTP stream.
>
> 0 1 2 3
> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> |V=2|P|X| CC |M| PT | sequence number |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | timestamp |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | synchronization source (SSRC) identifier |
> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
> | contributing source (CSRC) identifiers |
> | .... |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>
>
> Thanks and Regards,
>
> Srinivas G
>
>
> On Wed, Mar 29, 2023 at 6:28 AM Michael Wood <mornwood23 at gmail.com> wrote:
>
>>
>> On Tue, Mar 28, 2023 at 7:53 PM Michael Wood <mornwood23 at gmail.com>
>> wrote:
>>
>>> Yeah that is correct,
>>>
>>> I'm still unable to fix it, i'm trying to figure out what could cause
>>> the frame to not be properly decoded.
>>>
>>> Best,
>>> Milos
>>>
>>> On Sat, Mar 25, 2023 at 2:48 PM Strahinja Radman <dr.strashni at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On 25. 3. 2023., at 13:26, Michael Wood <mornwood23 at gmail.com> wrote:
>>>>
>>>> Hi Strahinja,
>>>>
>>>> Thanks for the help,
>>>>
>>>> I'm wondering if I need to update extradata (VPS,SPS,PPS) after initial
>>>> setup of the AVCodecContext from the initial value coming from RTP ?
>>>>
>>>> Best,
>>>> Milos
>>>>
>>>> On Sat, Mar 25, 2023 at 11:25 AM Strahinja Radman <
>>>> dr.strashni at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, 25 Mar 2023 at 11:09, Michael Wood <mornwood23 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 24, 2023 at 1:40 PM Strahinja Radman <
>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 24. 3. 2023., at 10:59, Michael Wood <mornwood23 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 24, 2023 at 10:28 AM Strahinja Radman <
>>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23. 3. 2023., at 20:19, Michael Wood <mornwood23 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 23, 2023 at 8:06 PM Strahinja Radman <
>>>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 23 Mar 2023 at 20:05, Michael Wood <mornwood23 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 23, 2023 at 6:20 PM Michael Wood <mornwood23 at gmail.com>
>>>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm having a weird issue with the decoding of H265 4K stream where
>>>>>>>> 1/3 of the frame is rendered normally and the other 2/3 is in green (
>>>>>>>> https://ibb.co/C8yFygm). This only happens if i try to decode 4K
>>>>>>>> stream, for non 4K streams everything works as expected.
>>>>>>>>
>>>>>>>> I'm using live555 library to get stream from RTSP server after
>>>>>>>> which packets are sent to the ffmpeg to be decoded and later rendered
>>>>>>>>
>>>>>>>> DESCRIBE response contains all required VPS,SPS,PPS fields that are
>>>>>>>> passed as extra data to the codec context.
>>>>>>>>
>>>>>>>> This is what is returned from DESCRIBE:
>>>>>>>>
>>>>>>>> v=0
>>>>>>>> o=- 1679598602413481 1679598602413481 IN IP4 192.168.1.212
>>>>>>>> s=Media Presentation
>>>>>>>> e=NONE
>>>>>>>> b=AS:5050
>>>>>>>> t=0 0
>>>>>>>> a=control:rtsp://
>>>>>>>> 192.168.1.212/Streaming/Channels/101/?transportmode=unicast&profile=Profile_1
>>>>>>>> m=video 0 RTP/AVP 96
>>>>>>>> c=IN IP4 0.0.0.0
>>>>>>>> b=AS:5000
>>>>>>>> a=recvonly
>>>>>>>> a=x-dimensions:3840,2160
>>>>>>>> a=control:rtsp://
>>>>>>>> 192.168.1.212/Streaming/Channels/101/trackID=1?transportmode=unicast&profile=Profile_1
>>>>>>>> a=rtpmap:96 H265/90000
>>>>>>>> a=fmtp:96 sprop-vps=QAEMA///AWAAAAMAgAAAAwAAAwCWAAC94Eg=;
>>>>>>>> sprop-sps=QgEDAWAAAAMAgAAAAwAAAwCWAACgAeAgAhx/jb33am7uS/YC3AQEBBAAAD6AAALuByHe5RAAmRgACsPgAEyMAAVh8EACZGAAKw+AATIwABWHwg==;
>>>>>>>> sprop-pps=RAHBcrCcCgFGZA==
>>>>>>>>
>>>>>>>> a=Media_header:MEDIAINFO=494D4B48010300000400050000000000000000000000000081000000000000000000000000000000;
>>>>>>>> a=appversion:1.0
>>>>>>>>
>>>>>>>> When trying to decode packets i'm getting this output from hevc,
>>>>>>>> the ffplay/VLC works normally with the incoming stream.
>>>>>>>>
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Format yuvj420p chosen by get_format().
>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 0.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>> temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 0.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>> 0, temporal_id: 0
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 1.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 2.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>> temporal_id: 2
>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>
>>>>>>>> If I write the stream to the file and play that file either through
>>>>>>>> ffplay or VLC the frames are rendered normally.
>>>>>>>>
>>>>>>>> Any suggestion would be welcomed.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Milos
>>>>>>>>
>>>>>>>> Hey Strahinja,
>>>>>>>>
>>>>>>>> Thanks for the answer,
>>>>>>>>
>>>>>>>> Can you provide more info on how to do it ?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Milos
>>>>>>>> _______________________________________________
>>>>>>>> Libav-user mailing list
>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>
>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>>>>>>> Hey Milos,
>>>>>>>>
>>>>>>>> In AVFrame structure, check width, height and stride variables.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Strahinja Radman
>>>>>>>> _______________________________________________
>>>>>>>> Libav-user mailing list
>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>
>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>>>>>>>
>>>>>>>> Hey Strahinja,
>>>>>>>>
>>>>>>>> It seems that both width & height are set correctly. I can't
>>>>>>>> find stride, i don't see any reference to it in the AVFrame header.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Milos
>>>>>>>> _______________________________________________
>>>>>>>> Libav-user mailing list
>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>
>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe”.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hey Milos,
>>>>>>>>
>>>>>>>> What does a parameter linesize say?
>>>>>>>>
>>>>>>>> AVFrame->linesize[0]
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Strahinja
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Hi Strahinja,
>>>>>>>
>>>>>>> The linesize looks good and corresponds to frame size
>>>>>>>
>>>>>>> frame->linesize[0] = 3840
>>>>>>> frame->linesize[1] = 1920
>>>>>>> frame->linesize[2] = 1920
>>>>>>>
>>>>>>> Best,
>>>>>>> Milos
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hey Milos,
>>>>>>>
>>>>>>> Can you use this method to save one frame to the PGM file and check
>>>>>>> its contents right after you receive an AVFrame
>>>>>>>
>>>>>>> static void pgm_save(unsigned char* buf, int wrap, int xsize, int
>>>>>>> ysize, const char* filename)
>>>>>>> {
>>>>>>> FILE* f;
>>>>>>> int i;
>>>>>>> f = fopen(filename, "w");
>>>>>>> fprintf(f, "P5\n%d %d\n%d\n", xsize, ysize, 255);
>>>>>>> for (i = 0; i < ysize; i++)
>>>>>>> fwrite(buf + i * wrap, 1, xsize, f);
>>>>>>> fclose(f);
>>>>>>> }
>>>>>>>
>>>>>>> buf is frame->data[0]
>>>>>>> wrap is frame->linesize[0]
>>>>>>> xsize is width and ysize is height
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Strahinja
>>>>>>>
>>>>>>>
>>>>>> Hi Strahinja,
>>>>>>
>>>>>> I'm getting the same result with saved picture, here is the link to
>>>>>> the output (
>>>>>> https://drive.google.com/file/d/1zEABl32ehVu-gnPdqjWciAb7im18VZIf/view?usp=sharing
>>>>>> )
>>>>>>
>>>>>> Best,
>>>>>> Milos
>>>>>>
>>>>>>
>>>>>
>>>>> Hey Milos,
>>>>>
>>>>> One thing to check is if live555 is configured correctly. Check if
>>>>> every decoder field is set properly and contains expected values. Sometimes
>>>>> stream properties can change during the decoding process and that can
>>>>> affect the output.
>>>>>
>>>>> Kind regards,
>>>>> Strahinja
>>>>>
>>>>>> --
>>>>>
>>>>> Regards
>>>>> Strahinja Radman
>>>>>
>>>>
>>>> Hey Milos,
>>>>
>>>> That shouldnt be necessary, that information should be used while
>>>> configuring the decoder context.
>>>>
>>>> Kind regards,
>>>> Strahinja
>>>>
>>>>
>>
>> Update:
>>
>> So I've managed to figure out what is happening, the NAL units I'm
>> receiving from the live555 library are actually fragments of a single
>> picture. Technically I need to combine those fragments into a single packet
>> and send it to ffmpeg for decoding.
>>
>> Any suggestions on how to recognize which fragments belong to a single
>> packet ?
>>
>> Best,
>> Milos
>> _______________________________________________
>> Libav-user mailing list
>> Libav-user at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/libav-user
>>
>> To unsubscribe, visit link above, or email
>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>
>
>
> --
> Thanks and Regards,
> Srinivas G
> _______________________________________________
> Libav-user mailing list
> Libav-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/libav-user
>
> To unsubscribe, visit link above, or email
> libav-user-request at ffmpeg.org with subject "unsubscribe".
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20230331/8bf17b2c/attachment.htm>
More information about the Libav-user
mailing list