[MPlayer-dev-eng] Yet another video API. :)

[Andriy N. Gritsenko]

    Hi!

    So after some discussions and rethinking whole thing, I've made this.
Big thanks for Rich and Ivan - this thing was partially based on Rich's
VP api, and many things from Ivan's letter (in December 2003) were very
useful. Images and buffers must be managed differently. My goals was:
1) have as little API as possible
2) use direct rendering as much as possible
3) use copying as little as possible
4) do memory allocations as little as possible
5) processing slices the same way as full frames (to be possible process
    slices even from decoder to vo without commiting to frame)
6) avoid pushing frames at all
So I mean I wanted to do all faster and with less memory.

    Rich asked me on IRC about how much API we need. Here is full list of
APIs which we need (AFAIK):
1) modules/config API (common);
2) stream API (file, network, etc.);
3) demuxer API;
4) video API;
5) audio API;
6) OSD/subtitles/input control API;
7) muxer API.

So with these APIs we will have those (sub)layers:
1) core layer (config parser, module finder/loader) - modules/config API;
2) input stream layer (file, cache, network, etc.) - stream API;
3) demuxer layer - from stream API to demuxer API;
4) video decoder layer - from demuxer API to video API;
5) audio decoder layer - from demuxer API to audio API;
6) subtitles layer - from stream or demuxer API to OSD/subtitles API;
7) video layer - video API (chains);
8) audio layer - audio API (chains);
9) video output layer - video API (end);
10) audio output layer - audio API (end);
11) video encoder layer - from video API to muxer API;
12) audio encoder layer - from audio API to muxer API;
13) muxer layer - from muxer API to stream API;

Each sublayer has own specifics and own number of APIs so don't have to
be messed with another. Frames that are pulled from layers 2...8 must be
pushed to layers 9...13 by application to do A/V sync.

Files in attachment contain video API (for using by layers 4,7,9,11, and
may be used by layer 6 too). Some minor parts are still not developed
yet, I'm still in doubt about them, see my comments there. :)

About future of G2... I think it's good to migrate from G1 to G2 and it
may be good to migrate step by step, i.e.: a) create new development
branch of MPlayer (1.1 for example); b) implement modules/config API to
it; c) create new demuxer layer with new demuxer API; d) develop new
video layer with new video API and so on. Each step will take some time
but it's much better than rewrite all at once.

    With best wishes.
    Andriy.

P.S. Don't be mad at my poor English, please. :)
P.P.S. All above is IMHO. :)
-------------- next part --------------




Video pipeline

Video pipeline consists of a collection of decoders, filters, output
devices, and encoders (vd/vf/vo/ve), collectively referred to as
nodes. All can either serve as a source of images, or a destination,
and some (filters) can serve as both. The nodes of the pipeline are
connected by links. A link maintains the negotiated parameters for
passing images between nodes: image format, dimensions, aspect,
strides, etc., and manages the buffers that will pass between its
endpoints. Most filters have exactly one input and one output link,
but the possibility is allowed for a filter to have multiple inputs or
multiple outputs.




Images

The link serves as the broker for images that will pass between its
source and destination nodes. All image structures (mp_image_t) are
associated with a link, and may only be used in the context of that
link. A filter cannot take a source image it receives and pass the
same image along as output to other parts of the pipeline!

However, via EXPORT and DIRECT type images, buffers can be passed on
in either direction in the chain. Knowing how to make use of them
allows codec and filter authors to eliminate unnecessary copying of
images, maximizing performance.



Images & locking

One of the most fundamental parts of the G2 video pipeline layer is
the system of buffer handling.

Each image has a reference count, or number of locks held on it. When
an image is obtained from vp_get_image, it initially has one lock.
Subsequent locks can be placed by either the source or destination
node associated with the link. Once the lock count reaches zero, the
image is no longer valid and must not be used. When this happens, the
owner of the image (if any) will be notified.

By means of locking, codecs may keep any number of reference frames
needed for decoding predicted frames, and temporal filters may keep
their input frames without having to make copies.





Image buffer types

AUTO(matic): Allocated and managed by the vp layer. Automatically uses
the negotiated strides for the link. No owner.

DIRECT(rendering): Buffer pointers are filled in by the destination
node's get_buffer function. The destination node becomes the owner,
and will be informed via release_buffer when the reference count
reaches zero.

EXPORT: Buffer pointers are filled in by the source node after
requesting the image. Keep in mind that they need not point to a
codec-internal buffer. They might point to the buffers from an
AUTO-type image earlier in the pipeline, or (in some very fancy
multiple-vo setups :) a DIRECT-type buffer from another branch of the
pipeline. The source node which obtains the EXPORT-type buffer becomes
its owner and will be informed when it's released.

[Note that both EXPORT and DIRECT buffers are very useful for avoiding
excess copying between filters. EXPORT should not be thought of as a
backwards-compatibility type for old codecs, because it can do a lot
more than just that!]

DUMMY: used for (intentionally!) dropped frames. No buffers
whatsoever, only pts and perhaps some other metainformation. May be
used for indication of EOS (end of stream) - ???



Image buffer flags:

READABLE: The buffer may be read (without incurring penalty).

PRESERVE: The source node expects the destination node not to clobber
buffer contents as long as the buffer is valid. (It rarely makes sense
for a source to keep an image locked unless the preserve flag is set!)

REUSABLE: The source node may clobber the buffer contents when
rendering subsequent frames, so the destination cannot expect it to
remain valid. (It rarely makes sense for a destination to keep an
image locked if the reusable flag is set!)

INDIRECT: The image buffer is not directly addressible, and may only
be drawn via slices.



Image flags:

MF_SLICE: Set if image isn't full image but slice instead. Filters who
don't accept slices will anyway get a full frame because video layer
will commit all slices to frame for it.

MF_FINAL: Set if image is last slice of a frame so all slices before it
may be committed to full frame.

MF_DROP: Set if decoder or filter may find that: a) there is yet one
frame before pts; b) that frame wants to be dropped. So it marks
the frame as may be dropped and any filter except those who desires
all frames may just ignore it.

MF_DECODE_ORDER: Set by decoder to indicate that frame was pulled in
decode order.

MF_NOT_IN_ORDER: Set by decoder to indicate that frame was pulled in
decode order and is out of order. For example in standard MPEG sequence
1 4 2 3 7 5 6 if frames are pulled in decode order then frames 4 and 7
will get that flag.




Filters API has number of functions. Each function should be not called
directly but only via vp_* API calls.

Most common rule is "if you get an image buffer then you _MUST_ return
it to owner", i.e. each vp_get_image() or vp_pull_image() must be
followed by either vp_release_image() either returned from function.
It is also available to call vp_keep_image() to save you a copy of
image or vp_export_image() to save you an image descriptor but that
keeped image also must be followed by one of these functions later.
I hope that rule is simple. If you follow that rule then you'll never
get your image lost or get memory leak.
So, image that returned by one of:
    vp_get_image()    vp_keep_image()    vp_pull_image()    vp_export_image()
must be consumed by one of:
    vp_release_image()		vp_show_image()
or by return from pull_image or get_image functions.

Basic filters API should implement:

mp_image_t *pull_image(vp_link_t *link, long long pts, int mode, int flags);

Called when one of the filter's output links desires a frame. If the
filter or decoder can provide one, it should return an image. Mode may
be one of:

VPP_MODE_FRAME: Get nearest frame before pts

VPP_MODE_KEYFRAME: Get nearest keyframe before pts

VPP_MODE_SLICE: Get next slice for current pts if VPP_SEEK is not set.
Otherwise reposition stream and get first slice of nearest frame before
pts (may have weird effects!)
(Note: If some filter between decoder and caller of vp_pull_image()
doesn't support slices then work as in VPP_MODE_FRAME)

Value of flags is interpreting as follows:

VPP_DECODE_ORDER: Get frame in decoder order if supported. Slices will
be pulled always in decode order.
(Note: If some filter between decoder and caller of vp_pull_image()
doesn't accept decoder order then that flag will be seized)

VPP_CANSKIP: Set if decoder may skip frames up to nearest frame before
pts if there are any. It does the effect of '-hardframedrop' switch and
may corrupt the image.

VPP_CANDROP: Set if decoder or filter may set MF_DROP flag in image.

VPP_SEEK: If that flag is set and pts points to the same frame as on
previous call then decoder must just give out the same frame again
(regetting frame). Otherwise if that flag is set then reposition the
stream to near pts and get the frame.


This function is main for filter functionality so it's most complex and
important. But it's a good thing that video layer will make it simple
as much as possible.

When decoder or filter already has an image which meet request then it
will return it, else it will get next image from it's source(s). If it
need a buffer for processing it may get it from any link. If decoder
requested for full frame but have only slice it may return slice, then
video layer will render slice into frame itself. If filter may work
with frame it got from source without additional buffer then it should
process frame and then return it.
For example, filter which just counts valid frames may be such simple:

mp_image_t *pull_image(vp_link_t *link, long long pts, int mode, int flags)
{
  mp_image_t *img;

  if ((img = vp_pull_image(link->src->in, pts, mode, flags)) == NULL)
    return NULL;
  link->src->priv->count++;
  return img;
}


Decoder behavior:
1) if VPP_SEEK is set:
 a) previous image pts is equal to requested pts: return previous image;
 b) else reposition source and return a frame with MF_NOT_IN_ORDER flag;
2) next image pts is above of requested pts (previous frame continues):
 return NULL;
3) next image pts is below or equal of requested pts and above of pts
of previous request:
 a) image damaged: release buffer and return NULL;
 b) else process image;
4) next image pts is below of pts of previous request (there is extra
frame in queue):
 a) VPP_CANSKIP is set: don't process it but get next frame from source
  until frame's pts is above of pts of previous request;
 b) VPP_CANDROP is set: process a frame but set MF_DROP flag;
 c) else just process a frame.

Filter behavior:
1) VPP_CANDROP is unset: process image as usual;
2) MF_DROP is set: if filter wants to process the image then process
 it and export it;
3) VPP_CANDROP is set but MF_DROP is unset: process image as usual
 and may set MF_DROP in resulting image.

Decoder may return slice even if VPP_MODE_SLICE is unset. In that case
video layer will render that slice for next filter in pipeline itself.


VO and encoders API should implement:

int process_image(vp_node_t *node, mp_image_t *img);

???????


Advanced API for filters to implement:

int open(vp_node_t *node);

As with all modules in the G2 framework, the open function is called
to initialize a module instance after loading the module. If open() is
implemented then it may perform some initialization tasks. Returns 0
on failure or 1 on success.

void close(vp_node_t *node);

Free all resources allocated by the filter and return to pre-open
state. This includes closing all links in *xin and *xout lists, if
filter handles ones.

int query_format(vp_link_t *link, fmtdesc_t *fmt);

Report whether the filter can accept images in the specified format.
In deciding how to respond, the filter should consider both (a)
whether it can process images in the specified format, and (b) whether
subsequent filters will be able to process its output, if it uses the
requested input format. The return value should be:

          0, if the format causes no conflicts
          1, if this filter chain cannot process the format

A filter that does not implement query_format implicitly accepts any
format except compressed and special types!

int config(vp_link_t *link, int w, int h, int samp_w, int samp_h, fmtdesc_t *fmt);

???????


void reset(vp_node_t *node, config_data_t *cfg);

If implemented then reset function will apply new settings to current
filter instance. It should not perform any dropping data or unlinking
any link.



int get_buffer(vp_link_t *link, mp_image_t *img);

Used for direct rendering and/or slices, the get_buffer function may
be requested by one of link's filter to provide a buffer to render
into. Requested buffer may be DIRECT or EXPORT type. If the filter
can provide a buffer matching the requested buffer memory flags then
the filter should setup the buffer pointers in the image structure
passed to it, and store any private data it needs to keep track of
the buffer. If a suitable buffer is not available, get_buffer should
return 0 to indicate failure. Otherwise it should return 1 for
success. If get_buffer isn't implemented then video layer will
reexport buffer from next filter in pipeline.

void release_buffer(vp_node_t *self, bufdesc_t *buf);

When a buffer's lock count reaches zero, its owner's release_buffer
function is called. The image type may be EXPORT or DIRECT.

int request_link(vp_link_t *link, int src);

That function designed for filters that may (and/or must) have more
than one input or output stream. If implemented then it must do some
link management (sorting, etc.) at vp_node_t structure. If src is 0
then request_link was called to add output for node link->src. If
src is 1 then request_link was called to add source to node link->dst.
Returns 0 if no more links supported or 1 on success. If request_link
not implemented then video layer assumes filter may have only one
source and output links and do that management itself.

int close_link(vp_link_t *link, int src);

That function designed for filters that may (and/or must) have more
than one input or output stream. If implemented then it must do some
link management (sorting, etc.) at vp_node_t structure. If src is 0
then close_link was called to remove output for node link->src. If
src is 1 then close_link was called to remove source from node
link->dst. Returns 0 if no such link removed or 1 on success. If
close_link not implemented then video layer assumes filter may have
only one source and output links and do that management itself.

















/* main vp API - use it instead of direct access to vp_node_s */

vp_node_t *vp_open(module_info_t *module, config_data_t *cfg);

As with all modules in the G2 framework, the open function is called
to initialize a module instance after loading the module. It should
setup all node structure setup, including startup settings, and may
perform other initialization tasks. Function may return NULL in case
of failure of initialization.

void vp_close(vp_node_t *self);

Free all resources allocated by the filter, destroy all links and
return to pre-open state. This includes converting any locked images
to AUTO type.

vp_link_t *vp_request_link(vp_node_t *dst, vp_node_t *src);

Try to link output of node src and input of node dst together. Returns
newly created link or NULL on failure.

void vp_close_link(vp_link_t *link);

Destroy link and release all images that belong to it.

void vp_reset(vp_node_t *node, config_data_t *cfg);

Apply new settings to current filter instance.

mp_image_t *vp_pull_image(vp_link_t *link, long long pts, int mode, int flags);

Really does the same as pull_image filter function, see description
above. But functionality is extended: vp_pull_image() does export
implicitly so returned image always belongs to link; it also compose
slices into frame if decoder returns slice instead of frame.

int vp_query_format(vp_link_t *head, fmtdesc_t *fmt, int force);

Report whether the filter chain can accept images in the specified
format. If there was a problem and force is set then function tries
to resolve it by inserting transcoding filter into conflicting link.
The return value should be:

          0, if the format causes no conflicts
         >0, if this chain cannot process the format

int vp_config(vp_link_t *link, int w, int h, int samp_w, int samp_h, fmtdesc_t *fmt);

???????

mp_image_t *vp_get_image(vp_link_t *link, int type, int flags, int x, int y, int w, int h);

Used for direct rendering and/or slices, the vp_get_image function
requests one of link's filter to provide a buffer for the caller to
render into. See description above about type and flags for buffer.
If a suitable buffer is not available then vp_get_image() should
return NULL. Otherwise it should return pointer to image structure.

void vp_release_image(mp_image_t *img);

Decrement reference count for image. If no references left then image
and buffer should be freed.

int vp_show_image(vp_node_t *node, mp_image_t *img);

???????

/* additional vp API */
mp_image_t *vp_keep_image(mp_image_t *img);

This should be used by filters that want to keep their input images
beyond subsequent requests, which could clobber the input if it
lies in a reusable buffer.

mp_image_t *vp_export_image(vp_link_t *link, mp_image_t *orig);

Export current image to link, i.e. create new image descriptor
belonging to link from orig descriptor and return new descriptor.
That function helps when you want preserve your descriptor and
buffer for later. Note that buffer may be overwritten by following
filters so if you want keep buffer contents too then you should
use vp_keep_image() instead. So if your pull_image function has:
  return img;
then you'll lost that img descriptor but if instead it'll have:
  return vp_export_image(link, img);
then your img will be preserved for you.

void vp_select_stride(int *stride, stride_rest_t *r, fmtdesc_t *fmt);

???????

-------------- next part --------------



#include "cfg.h"

// filter capabilities/limitations
#define	VP_CAPS_ACCEPT_SLICES	(1<<16)	/* if filter accepts slices */
#define	VP_CAPS_NO_DECODE_ORDER	(1<<17)	/* if filter doesn't accept decode order */

typedef struct vp_node_s vp_node_t;
typedef struct vp_priv_s vp_priv_t;

#include "imgfmt.h"




// BUFFER DESCRIPTOR
// Stores layout of the actual buffers and nothing else.
// Often this can rereferenced when reexporting or re-dr'ing.
// Owner is node which allocated a buffer, if there is any
// Buffer management types
#define BUF_TYPE_DUMMY      0 // no buffers needed (dropped frame or eos)
#define BUF_TYPE_AUTO       1 // auto-managed by link layer (internal purpose?)
#define BUF_TYPE_EXPORT     2 // exported from source to dest
#define BUF_TYPE_DIRECT     3 // exported from dest to source (DR)
//#define BUF_TYPE_INDIRECT   4 // indirect rendering via slices, no pointers
// Memory restrictions
#define BUF_MEM_READABLE  0x1 // buffer is in readable memory
#define BUF_MEM_PRESERVE  0x2 // dest will not clobber buffer contents
#define BUF_MEM_REUSABLE  0x4 // source can reuse & re-return buffer >1 times
typedef struct {
	int type;
	int flags;
	vp_node_t *owner;	/* who allocated this buffer */
	fmtdesc_t *fmt;
	int lock;		/* reference count */
	int w[MAX_PLANES];
	int h[MAX_PLANES];
	int stride[MAX_PLANES];	/* size of pixel */
	int size[MAX_PLANES];
	unsigned char *planes[MAX_PLANES];
	unsigned char *palette;
	void *priv;            // private data for owner
} bufdesc_t;




// META-FRAME
// Describes the properties of an image which pertain to it being
// a frame of a movie. Often this can be copied as-is when only
// filtering the contents of the image.
#define	MF_SLICE	(1<<0)	/* it's not full frame but slice */
#define	MF_FINAL	(1<<1)	/* it's last slice or EOF */
#define	MF_DROP		(1<<2)	/* set to 1 if frame is out of pts */
#define MF_DECODE_ORDER	(1<<3)	/* if frame is in decode order */
#define	MF_NOT_IN_ORDER	(1<<4)	/* if frame isn't in display order */
#define	MF_EOS		(1<<5)	/* set by decoder when no images left */
typedef struct {
	long long pts;     // absolute pts of this frame, -1 if unknown
	int rel_pts;       // pts relative to previous frame (required)
	int duration;      // duration of frame, -1 if unknown
	int color_type;
	int pict_type;
	int field_based;
	int field_flags;
	int x, y;		// disposition of slice
	int w, h;		// size of slice
	int flags;
} imagedesc_t;


// MPLAYER IMAGE
// This is the main image structure used for passing frames between
// nodes of the pipeline. Each image is always associated with a link.
// The elements of mp_image_t are divided into substructs according
// to whether they are properties of the frame the image represents,
// or properties of the buffer in which it is stored. This allows one
// or the other to easily be copied without modification.

typedef struct {
	vp_link_t *link;
	imagedesc_t mf;
	bufdesc_t *buf;		// pointer to buffer for image
	bufdesc_t bb;		// builted-in buffer, may be unused
} mp_image_t;




// STRIDE RESTRICTIONS
#define STRIDE_ALIGN         0x01
#define STRIDE_EXACT         0x02
#define STRIDE_STATIC        0x04
#define STRIDE_POSITIVE      0x08
#define STRIDE_NEGATIVE      0x10
#define STRIDE_COMMON        0x20
#define STRIDE_COMMON_UV     0x40
#define STRIDE_COMMON_PLANE  0x80
typedef struct {
	int flags;
	int align[MAX_PLANES];
	int stride[MAX_PLANES];
} stride_rest_t;




typedef struct {
	vp_node_t *src, *dst;
	fmtdesc_t *fmt;
	int w, h;
	int samp_w, samp_h;
	long long last_pts;
	stride_rest_t sr;
} vp_link_t;


typedef struct {
	// Main vp api
	int (*open)(vp_node_t *node);
	void (*close)(vp_node_t *node);
	void (*reset)(vp_node_t *node, config_data_t *cfg);
	// if called for dest node then src==0, otherwise - for src node
	int (*request_link)(vp_link_t *link, int src);
	int (*close_link)(vp_link_t *link, int src);
	mp_image_t *(*pull_image)(vp_link_t *link, long long pts, int mode, int flags);
	int (*query_format)(vp_link_t *link, fmtdesc_t *fmt);
	int (*config)(vp_link_t *link, int w, int h, int samp_w, int samp_h, fmtdesc_t *fmt);
	// Direct rendering
	int (*get_buffer)(vp_link_t *link, mp_image_t *img);
	void (*release_buffer)(vp_node_t *self, bufdesc_t *buf);
	// Slices
//	void (*attach_slices)(vp_link_t *link, mp_image_t *img);
//	void (*detach_slices)(vp_node_t *self, mp_image_t *img);
//	void (*draw_slice)(vp_link_t *link, mp_image_t *img,
//		unsigned char **src, int *stride, int x, int y, int w, int h);
//	void (*commit_slice)(vp_link_t *link, mp_image_t *img, int x, int y, int w, int h);
	// VO API
	int (*process_image)(vp_node_t *node, mp_image_t *img);
} vp_functions_t;

struct vp_node_s {
	module_info_t *info;
	config_data_t *cfg;
	vp_functions_t *func;
	// Links, primary & extra
	vp_link_t *in, *out;
	vp_link_t **xin, **xout;	// must be handled only by filter!
	mp_image_t *pending;
	// Private data for this node of the pipeline
	vp_priv_t *priv;
};



/* flags for vp_pull_image() */
#define	VPP_MODE_FRAME		0	/* get nearest frame */
#define VPP_MODE_KEYFRAME	1	/* get nearest keyframe */
#define VPP_MODE_SLICE		2	/* get next slice */

#define VPP_DECODE_ORDER (1<<0)	/* get frame in decoder order */
#define VPP_CANSKIP	(1<<1)	/* decoder can skip frame (-hardframedrop) */
#define VPP_CANDROP	(1<<2)	/* filter/decoder may set MF_DROP */
#define VPP_SEEK	(1<<3)	/* seek or reget the same frame */

/* main vp API - use it instead of direct access of vp_node_s */
vp_node_t *vp_open(module_info_t *module, config_data_t *cfg);
void vp_close(vp_node_t *self);

void vp_reset(vp_node_t *node, config_data_t *cfg);

vp_link_t *vp_request_link(vp_node_t *dst, vp_node_t *src);
void vp_close_link(vp_link_t *link);

mp_image_t *vp_pull_image(vp_link_t *link, long long pts, int mode, int flags);
int vp_query_format(vp_link_t *head, fmtdesc_t *fmt, int force);
int vp_config(vp_link_t *link, int w, int h, int samp_w, int samp_h, fmtdesc_t *fmt);

mp_image_t *vp_get_image(vp_link_t *link, int type, int flags, int x, int y, int w, int h);
void vp_release_image(mp_image_t *img);

int vp_show_image(vp_node_t *node, mp_image_t *img);

/* additional vp API */
mp_image_t *vp_keep_image(mp_image_t *img);
mp_image_t *vp_export_image(vp_link_t *link, mp_image_t *orig);

void vp_select_stride(int *stride, stride_rest_t *r, fmtdesc_t *fmt);

//int vp_attach_slices(vp_link_t *link, mp_image_t *img);
//void vp_draw_slice(vp_link_t *link, mp_image_t *img, unsigned char **src, int *stride, int x, int y, int w, int h);
//void vp_commit_slice(vp_link_t *link, mp_image_t *src, int x, int y, int w, int h);

-------------- next part --------------


#include <stdlib.h>
#include <malloc.h>

#include "cfg.h"
#include "vp.h"


#define	IMAGE_POOL_SIZE	32	/* minimal pool size */

typedef struct image_pool {
    mp_image_t pool[IMAGE_POOL_SIZE];
    struct image_pool *next;
} image_pool_t;

static image_pool_t *image_pool = NULL;


static mp_image_t *vp_allocate_image(void)
{
    mp_image_t **pool = &image_pool;
    int i;

#define	image(x)	(*pool)->pool[x]
    while(*pool)
    {
	for (i = 0; i < IMAGE_POOL_SIZE; i++)
	{
	    if (!image(x).link && !image(x).pp.lock)
		return &image(x); // we check both ->link and pp.lock now since:
		// link==NULL and pp.lock!=0 when buffer still used by another link
		// link!=NULL and pp.lock==0 when this is exported image
	}
	pool = &(*pool)->next;
    }
    /* we cannot do realloc here, all references (i.e. all images)
       will be invalid after realloc :( */
    *pool = calloc(1, sizeof(image_pool_t));
    return &image(0);
#undef	image
}


vp_link_t *vp_request_link(vp_node_t *dst, vp_node_t *src)
{
    vp_link_t *link = malloc(sizeof(vp_link_t));

    if ((!dst->func->request_link && dst->in) ||
	(!src->func->request_link && src->out))
	return NULL;		// src or dst is already linked
    link = malloc(sizeof(vp_link_t));
    link->src = src;
    link->dst = dst;
    link->fmt = NULL;
    link->last_pts = -1;
    memset(&link->sr, 0, sizeof(stride_rest_t));
    if (dst->func->request_link)	// try to add link to destination node
    {
	if (!dst->func->request_link(link, 1))
	{
	    free(link);
	    return NULL;
	}
    }
    else
	dst->in = link;
    if (src->func->request_link)	// try to add link to source node
    {
	if (!src->func->request_link(link, 0))
	{
	    if (dst->func->close_link)
		dst->func->close_link(link, 1);
	    else
		dst->in = NULL;
	    free(link);
	    return NULL;
	}
    }
    else
	src->out = link;
    return link;
}


void vp_close_link(vp_link_t *link)
{
    mp_image_pool **pool = &image_pool;

    /* close the link */
    if (src->func->close_link)
	src->func->close_link(link, 0);
    else
	src->out = NULL;
    if (dst->func->close_link)
	dst->func->close_link(link, 1);
    else
	dst->in = NULL;
    /* release all images owned by link */
    while (*pool)
    {
	for (i = 0; i < IMAGE_POOL_SIZE; i++)
	    if ((*pool)->pool[x].link == link)
		vp_release_image(&(*pool)->pool[x]);
	pool = &(*pool)->next;
    }
    /* release memory */
    free(link);
}


vp_node_t *vp_open(module_info_t *module, config_data_t *cfg)
{
    vp_node_t *node;

    /* create an instance */
    node = malloc(vp_node_t);
    node->info = module;
    node->func = (vp_function_t *)module->entry;
    node->cfg = mpconfig_get_cfg(module, cfg);
    node->in = node->out = NULL;
    node->xin = node->xout = NULL;
    node->pending = NULL;
    node->priv = NULL;
    /* setting up the instance */
    if (node->func->open && !node->func->open(node))
    {
	mpconfig_release_cfg(module, node->cfg);
	free(node);
	return NULL;
    }
    return node;
}


void vp_reset(vp_node_t *node, config_data_t *cfg)
{
    if (!cfg) return;		// nothing to do?
    if (node->func->reset)	// let filter do it nice
	node->func->reset(node, cfg);
    else			// it cannot do it so just replace config data
    {
	mpconfig_release_cfg(node->info, node->cfg);
	node->cfg = mpconfig_get_cfg(node->info, cfg);
    }
}


void vp_close(vp_node_t *node)
{
    int i;
    mp_image_pool **pool = &image_pool;
    bufdesc_t *buf;
    unsigned char *planes[MAX_PLANES];
    int size;

    /* does it has buffers in pool? if so then, sorry, replace them with AUTO */
    while (*pool)
    {
	for (i = 0; i < IMAGE_POOL_SIZE; i++)
	{
	    buf = &(*pool)->pool[x].bb;
	    if (buf->lock && buf->owner == node)
	    {
		size = 0;
		for (i = 0; i < buf->fmt->num_planes; i++) size += buf->size[i];
		planes[0] = memalign(64, size);
		for (i = 1; i < buf->fmt->num_planes; i++)
		    planes[buf->fmt->order[i]] = planes[i-1] + buf->size[i-1];
		memcpy_pic_planes(planes, buf->planes, buf->w, buf->h,
				buf->stride, buf->stride, buf->fmt);
		if (node->release_buffer)
		    node->release_buffer(node, buf);
		memcpy(&buf->planes, &planes, MAX_PLANES*sizeof(unsigned char *));
		buf->owner = NULL;
		buf->type = BUF_TYPE_AUTO;
	    }
	}
	pool = &(*pool)->next;
    }
    if (node->func->close)
	node->func->close(node);	// it handles xin and xout lists too!
    if (node->in)
	vp_close_link(node->in);
    if (node->out)
	vp_close_link(node->out);
    mpconfig_release_cfg(node->info, node->cfg);
    free(node);
}


#if 0
////int vp_config(vp_link_t *link, int w, int h, int samp_w, int samp_h, fmtdesc_t *fmt);
int vp_config(vp_link_t *link, fmtdesc_t *fmt,
	int w, int h, int samp_w, int samp_h, fmtdesc *fmt)
{
	vp_node_t *src = link->src, *dest = link->dest;
	if (!dest->query_format(link, fmt)) {
		vp_insert_filter(link, vp_open_filter(vp_find_filter("scale"), NULL));
		dest = link->dest;
	}
	if (!dest->config(link, fmt, w, h, samp_w, samp_h, time_n, time_d))
		return 0;
	link->fmt = fmt;
	link->w = w;
	link->h = h;
	link->samp_w = samp_w;
	link->samp_h = samp_h;
	link->time_n = time_n;
	link->time_d = time_d;
	return 1;
}



// Before calling, the stride restrictions must already be checked for
// consistency! Otherwise the results are undefined! The stride array
// should be initialized with width or desired stride.
////void vp_select_stride(int *stride, stride_rest_t *r, fmtdesc_t *fmt);
void vp_select_stride(int *stride, stride_rest_t *r, fmtdesc_t *fmt)
{
	int s[MAX_PLANES];
	if (r->flags & STRIDE_EXACT) {
		for (i=0; i<fmt->num_planes; i++)
			stride[i] = r->stride[i];
		return;
	}
	for (i=0; i<fmt->num_planes; i++)
		s[i] = abs(stride[i]);
	for (i=0; i<fmt->num_planes; i++) {
		if (r->flags & STRIDE_EXACT)
			s[i] = r->stride[i];
		if (r->flags & STRIDE_ALIGN && s[i]%r->align[i])
			s[i] += r->align[i] - s[i]%r->align[i];
		if (r->flags & STRIDE_COMMON_STRIDE)
			s[i] = s[0] >> fmt->ssx[i];
	}
	for (i=0; i<fmt->num_planes; i++) {
		if ( !(r->flags & STRIDE_POSITIVE)
			&& (stride[i] < 0 || (r->flags & STRIDE_NEGATIVE)) )
			stride[i] = -s[i];
		else
			stride[i] = s[i];
	}
}
#endif


mp_image_t *vp_get_image(vp_link_t *link, int type, int flags,
				int x, int y, int w, int h)
{
    fmtdesc_t *fmt = link->fmt;
    vp_link_t *dstlink = link;
    mp_image_t *img;
    bufdesc_t *buf;
    int size;

    /* check for get_buffer and try to get thru chain */
    if (type == BUF_TYPE_DIRECT)
	while (!link->dst->get_buffer)
	{
	    if (!link->dst->out) return NULL;
	    link = link->dst->out;
	}
    else if (type == BUF_TYPE_EXPORT)
	while (!link->src->get_buffer)
	{
	    if (!link->src->in) return NULL;
	    link = link->src->in;
	}
    /* some initial setup for buffer */
    img = vp_allocate_image();
    buf = img->buf = &img->bb;
    img->mf.x = x;
    img->mf.y = y;
    img->mf.w = w;
    img->mf.h = h;
    img->mf.pts = -1;
    img->mf.duration = -1;
    buf->type = type;
    buf->flags = flags;
    buf->fmt = fmt;
    for (i = 0; i < fmt->num_planes; i++)
    {
	buf->w[i] = w >> fmt->ssx[i];
	buf->h[i] = h >> fmt->ssy[i];
	buf->stride[i] = link->stride[i];
	buf->size[i] = buf->h[i]*abs(buf->stride[i]);
    }
    switch (type)
    {
	case BUF_AUTO:
	    size = 0;
	    for (i = 0; i < fmt->num_planes; i++) size += buf->size[i];
	    buf->planes[0] = memalign(64, size);
	    // order planes in "common_plane" layout
	    for (i = 1; i < fmt->num_planes; i++)
		buf->planes[fmt->order[i]] = buf->planes[i-1] + buf->size[i-1];
	    for (i = 0; i < fmt->num_planes; i++)
		memset(buf->planes[i], fmt->black[i], buf->size[i]);
	case BUF_DUMMY:
	    buf->owner = NULL;
	    break;
	case BUF_EXPORT:
	    buf->owner = link->src;
	    if (!link->src->func->get_buffer(dstlink, img))
		return NULL;
	    break;
	case BUF_DIRECT:
	    buf->owner = link->dst;
	    if (!link->dst->func->get_buffer(dstlink, img))
		return NULL;
	    break;
	default:		// unknown image type!
	    return NULL;
    }
    img->link = dstlink;
    buf->lock++;
    return img;
}


// This should be used by filters that want to keep their input images
// beyond subsequent requests, which could clobber the input if it
// lies in a reusable buffer.
mp_image_t *vp_keep_image(mp_image_t *img)
{
    mp_image_t *copy;

    img->buf->lock++;
    if (!(img->buf->flags & BUF_MEM_REUSABLE))
	return img;
    copy = vp_get_image(img->link, BUF_TYPE_AUTO, 0, img->mf.x, img->mf.y,
			img->mf.w, img->mf.h);
    memcpy_pic_planes(copy->buf->planes, img->buf->planes, img->mf.w, img->mf.h,
		      copy->buf->stride, img->buf->stride, img->link->fmt);
    vp_release_image(img);
    return copy;
}


mp_image_t *vp_export_image(vp_link_t *link, mp_image_t *orig)
{
    mp_image_t *img = vp_allocate_image();

    if (orig->buf->fmt != link->fmt)	// ugly filter!!!
    {					// cannot just copy between formats!!!
	mp_error("frame transferred to another format, dropped!");
	return NULL;
    }
    img->buf = orig->buf;	// buffer is the same
    img->buf->lock++;		// but mark it used one more time
    img->link = link;
    memcpy(&img->mf, &orig->mf, sizeof(imagedesc_t));	// just copy it
    return img;
}


void vp_release_image(mp_image_t *img)
{
    bufdesc_t *buf = img->buf;

    img->link = NULL;		// it's unused now
    if (--buf->lock) return;
    if (buf->owner)		// BUF_TYPE_EXPORT and BUF_TYPE_DIRECT
    {
	if (buf->owner->func->release_buffer) // hmm, it doesn't worry to deallocate?
	    buf->owner->func->release_buffer(buf->owner, buf);
	buf->owner = NULL;
	return;
    }
    if (buf->type != BUF_TYPE_AUTO) return;
    free(buf->planes[0]);	// deallocate memory
}


int vp_query_format(vp_link_t *link, fmtdesc_t *fmt, int force)
{
    if (!link || !link->dst) return 0;	// dead end, assume it's ok
    if (!link->dst->func->query_format)	// assume it accepts any format
	return vp_query_format(link->dst->out, fmt, force);
    if (!link->dst->func->query_format(link, fmt))
	return 0;
    if (!force) return 1;
    ??????? // try auto-insert format converting filter before link->dst
}


////int vp_show_image(vp_node_t *node, mp_image_t *img);


#if 0
static int vp_attach_slices(vp_link_t *link, mp_image_t *img)
{
	vp_node_t *dest = link->dest;
	mp_image_t *img2;
	if (img->buf.type != BUF_TYPE_AUTO && img->buf.type != BUF_TYPE_EXPORT)
		return 0;
	if (dest->attach_slices)
		return dest->attach_slices(link, img);
	img2 = vp_get_image(link, BUF_TYPE_INDIRECT, 0);
}


static void vp_draw_slice(vp_link_t *link, mp_image_t *img, unsigned char **src, int *stride, int x, int y, int w, int h)
{
	if (img->buf.type == BUF_TYPE_DIRECT)
		memcpy_subpic_planes(img->buf.planes, src, x, y, 0, 0,
			w, h, img->buf.stride, stride, img->fmt);
	if (link->dest->draw_slice)
		link->dest->draw_slice(link, img, src, stride, x, y, w, h);
}

static void vp_commit_slice(vp_link_t *link, mp_image_t *src, int x, int y, int w, int h)
{
	if (src->target->buf.type == BUF_TYPE_DIRECT)
		memcpy_subpic_planes(src->target->buf.planes, src->buf.planes,
			x, y, 0, 0, w, h, src->target->buf.stride,
			src->buf.stride, dest->fmt);
	if (link->dest->commit_slice)
		link->dest->commit_slice(link, src, x, y, w, h);
	else if (link->dest->draw_slice) {
		int i;
		unsigned char *src2[MAX_PLANES];
		for (i=0; i<link->fmt->num_planes; i++)
			src2[i] = src->buf.planes[i]
				+ src->buf.stride[i]*(y>>link->fmt->ssy[i])
				+ ((link->fmt->bpp[i]*(x>>link->fmt->ssx[i]))>>3);
		link->dest->draw_slice(link, dest, src2, src->buf.stride, x, y, w, h);
	}
}
#endif



mp_image_t *vp_pull_image(vp_link_t *link, long long pts, int mode, int flags)
{
    mp_image_t *img;

    /* getting the same image again? may be recursion? avoid it! */
    if (pts == link->last_pts && !(flags & VPP_SEEK))
	return NULL;
    /* I think it's impossible but... */
    if (!link->src || !link->src->func->pull_image)
	return NULL;
    /* erasure for flags in dependence of filter capabilities */
    if (link->dst && !(link->dst->info->flags & VP_CAPS_ACCEPT_SLICES))
	mode &= ~VPP_MODE_SLICE;
    if (link->dst && (link->dst->info->flags & VP_CAPS_NO_DECODE_ORDER))
	flags &= ~VPP_DECODE_ORDER;
    /* call function and return if no image ready yet */
    if ((img = link->src->func->pull_image(link, pts, mode, flags)) == NULL)
	return NULL;
    /* if it's a slice but slice wasn't requested then commit seq. slices */
    if ((img->mf.flags & MF_SLICE) && mode != VPP_MODE_SLICE)
    {
	mp_image_t *img2;

	if (!(img2 = vp_get_image(link, BUF_TYPE_DIRECT, 0, 0, 0, link->w, link->h)))
	    img2 = vp_get_image(link, BUF_TYPE_AUTO, 0, 0, 0, link->w, link->h);
	while (img)
	{
	    /* commit slice - I'm afraid it's never possible to do it 
		through filter chain so let's just copy it */
	    memcpy_subpic_planes(img2->buf->planes, img->buf->planes,
				x, y, 0, 0, w, h, img2->buf->stride,
				img->buf->stride, link->fmt);
	    vp_release_image(img); // line below is valid until multithreaded
	    if ((img->mf.flags & MF_SLICE) && !(img->mf.flags & MF_FINAL))
		img = link->src->func->pull_image(link, pts, mode, flags);
	    else
		break;
	}
	img = img2;
    }
    /* if image was completed then check some fields and return it */
    if (img->buf->fmt != link->fmt)	// ugly filter!!!
    {					// cannot just copy between formats!!!
	vp_release_image(img);
	mp_error("frame transferred to another format, dropped!");
	return NULL;
    }
    link->last_pts = img->mf.pts;
    img->link = link;
    return img;
}