[Libav-user] achieving greater than 15 fps playback

[Blake Senftner]

Hello,

Working on video (no audio) playback library, I am only seeing frame rates of 15 fps or less. My code started based on the Dranger examples, but has since been updated to the latest APIs (such that I compile with no depreciated warnings.) I am using Visual Studio 2013 Community. 

I have a 160x40 29.97 fps h264 clip with timecodes, and it plays back at exactly half speed, consuming very little cpu. An rtsp h264 stream from an IP camera plays back between 10 & 11 fps, while the latest full HD resolution Rogue One trailer plays back at 15 fps (both only consuming < 20% CPU.) 

My code does not use SDL, so I am using neosmart cross platform pevents (https://github.com/neosmart/pevents). 

I am using one thread to read packets, decode them to yuv & convert to RGB. OpenGL does the display in the main thread.

Does achieving higher frame rates require the packet reading and packet decoding in separate threads? I thought the 160x40 file would decompress fast enough that I’d need to add necessary timing logic to delay the frames, but that is not the case. It plays

Here’s the general logic of my playback setup, abbreviated for clarity, and the packet reading is below that:

------------------------------------------------------------------------------------------
avformat_open_input( &mp_format_context, [path to file or stream], NULL, NULL );
avformat_find_stream_info( mp_format_context, NULL );
m_video_stream = -1;
for (ce_uint i = 0; i < (ce_uint)mp_format_context->nb_streams; i++) {
	if (mp_format_context->streams[i]->codecpar->codec_type==AVMEDIA_TYPE_VIDEO) {
		m_video_stream = i;
		break;
	}
}
if (m_video_stream == -1) 
	return false;

AVCodecParameters* codecPars = mp_format_context->streams[m_video_stream]->codecpar;

AVCodec* pCodec = avcodec_find_decoder( codecPars->codec_id );
if (!pCodec) 
	return false;

mp_codec_context = avcodec_alloc_context3( pCodec );
if (avcodec_parameters_to_context( mp_codec_context, codecPars ) != 0)
	return false;
	
mp_codec_context->thread_count = 0; // tried 4, 8… makes no difference

AVDictionary* codec_options = NULL;
av_dict_set( &codec_options, "threads", "auto", 0 );	// makes no difference, tried 4 & 8 too
if (m_stream_type == 2)
	av_dict_set( &codec_options, "rtsp_transport", "tcp", 0 ); // needed for ffplay to work with my ip cam, makes no difference here...

if (avcodec_open2( mp_codec_context, pCodec, &codec_options ) < 0)
	return false;

mp_decompressed_frame = av_frame_alloc();
mp_display_frame = av_frame_alloc();
if (mp_decompressed_frame == NULL || mp_display_frame == NULL)
	return false;
	
ce_int numBytes = av_image_get_buffer_size( AV_PIX_FMT_RGBA, mp_codec_context->width, mp_codec_context->height, 1 );
mp_display_buffer = (uint8_t *)av_malloc( numBytes*sizeof(uint8_t) );
av_image_fill_arrays( mp_display_frame->data, mp_display_frame->linesize,		
		mp_display_buffer, AV_PIX_FMT_RGBA,
		mp_codec_context->width, mp_codec_context->height, 1 );

mp_sws_context = sws_getContext( mp_codec_context->width, mp_codec_context->height, mp_codec_context->pix_fmt,
																	 mp_codec_context->width, mp_codec_context->height, AV_PIX_FMT_RGBA,
																	 SWS_BILINEAR, NULL, NULL, NULL );

—————————————————————————————————————————

In my packet reading thread, the logic is simple:

——————————————————————————————————
uint64_t milliseconds = 1;

bool media_has_ended = false;
bool camera_has_terminated = false;
ce_uint packet_errors = 0;

while (true) {
	// this basically allows the thread to idle (sleep) but it will
	// wake up if the events are signaled
	dwWaitRes = neosmart::WaitForEvent(m_stop_local_spin_event, milliseconds);

	if (0 == dwWaitRes) // stop event
		break;
		
	else if (m_is_playing && !m_paused) {

		// I’ve got 16 packet buffers so I can look at their data, but only use one at a time: 
		ce_uint curr_packet_index = m_total_video_packets % CE_LIBAV_NUMVIDPKTS;
		AVPacket* curr_packet = &m_packet[ curr_packet_index ];
		m_total_video_packets++;

		int stream_status = av_read_frame(mp_format_context, curr_packet);
		if (stream_status < 0) {

			if (m_stream_type == 0) { // end of media file // stream_types: 0=Media, 1=USB, 2=IP
				media_has_ended = true;
				break;
			}
			else { // camera stream has terminated unexpectedly 
				camera_has_terminated = true;
				break;
			}
		}
		else { // stream delivered a packet fine 
			
			if (curr_packet->stream_index == m_video_stream) {
				
				// Decode video frame(s):
				int ret = avcodec_send_packet( mp_codec_context, curr_packet );
				if (ret < 0) {
					
					if (ret == AVERROR_EOF && m_stream_type == 0) {
						media_has_ended = true;
						break;
					}
					else packet_errors++;
				}
				else {
					
					while (!ret)
					{
						ret = avcodec_receive_frame( mp_codec_context, mp_decompressed_frame );
						if (!ret)
							handle_new_frame( curr_packet, curr_packet_index );
						
					}
	
				}
				av_packet_unref(curr_packet);
			
		}		// end stream delivered a packet

	}			// end if (m_is_playing && !m_paused)
}				// end while (true)

if (media_has_ended) {
	if (mp_stream_ended_callback) 
			(mp_stream_ended_callback)(m_frames_received, mp_stream_ended_object);
	
if (camera_has_terminated)
	if (mp_term_callback) 
			(mp_term_callback)(mp_term_object);
	

——————————————————————————————————


Sincerely,
-Blake Senftner
Mad Computer Scientist

p.s.

My ffmpeg libav* 3.2 libs are, downloaded pre-built from Zeranoe early November, my versions are:

libavutil      55. 34.100 / 55. 34.100
libavcodec     57. 64.100 / 57. 64.100
libavformat    57. 56.100 / 57. 56.100
libavdevice    57.  1.100 / 57.  1.100
libavfilter     6. 65.100 /  6. 65.100
libswscale      4.  2.100 /  4.  2.100
libswresample   2.  3.100 /  2.  3.100
libpostproc    54.  1.100 / 54.  1.100