[MPlayer-dev-eng] Documenting and extending the subtitles system

[Uoti Urpala]

On Mon, 2009-02-23 at 16:07 +0100, Nicolas George wrote:
> There are currently two incompatible mechanisms to overlay something on the
> video: OSD (the old one) and EOSD (the more recent one).

Reimar already commented on some points, I won't repeat those.

It's not clear whether you're considering only the render-on-video part
or also how OSD content state is maintained and updated. Those parts are
separate; the code doing the rendering need not have any knowledge about
how OSD internally maintains its list of objects. Your comments about
the current mechanisms generally fail to distinguish the API for
updating OSD state and the API for renderers, which are fairly
independent.

> * OSD dates back to 2001 without major change. Its features are:

>   - Overlaid stuff is stored as an almost-static list of objects in a global
>     variable.
> 
>   - Overlaid objects have a type field describing their origin. For each
>     displayed frame, the OSD system loops on all origins, queries global
>     variables and updates the objects.
>
>   - The origin field also implies the format of the OSD object, and thus the
>     way to render it:
> 
>     - SPU objects are rendered by a call to the spudec subsystem;
> 
>     - text objects are in fact made of a gray map (1 octet per pixel) plus a
>       bitmap mask (1 octet by pixel as a boolean).

This stuff is all related to maintaining OSD state and is not visible to
the part doing the rendering.

> Now for possible evolutions. I think that EOSD is almost suitable to use for
> all OSD, but two things are missing: full color support and separation from
> libass.

Now this comment clearly only considers the rendering part.

> For full color support, the changes could be pretty small: the EODS objects
> are currently an alpha mask and an unique color, it would be enough to
> replace the unique color by a complete color image.
> 
> If this color image is stored in the target color space, and the alpha
> channel is prescaled for all the subsamplings of the pixel formats, it would
> probably be just as efficient as the current version, and support all pixel
> formats automatically.

The main problem with this is that it's worse for caching rendered
fonts. Any color change will require rendering a separate fullcolor
bitmap. It'll also use more memory for caching. I think the main
alternatives are:
1) Cache fullcolor bitmaps. Uses more memory and requires rendering from
scratch more often than current model.
2) Cache alpha bitmaps but export fullcolor bitmaps from libass, doing
the alpha->fullcolor compositing inside libass (possibly caching that
for a shorter time).
3) Export alpha bitmaps and allow renderers to implement a possibly more
efficient compositing step (requiring more complicated compositing
primitives than current EOSD).


> As for separating EOSD from libass, this is not much either, and it would
> lead to a much cleaner design anyway. What we need is a well-isolated EOSD
> subsystem with:

I think "EOSD" is best used to refer to the renderer API. You're
considering other related functionality here.

> - an input API, for anyone (libass among others) to add or remove EOSD
>   objects;

Yes.

> - a callback API, for anyone to get notification of significant events (such
>   as timestamps expiration);

This isn't so obvious. Currently there is no such API and I see no clear
need for one. Subtitle events are probably best handled by the subtitle
"codec". OSD objects may have expiration times, but it doesn't seem
necessary to have those "in the OSD API".

> - an output API, for the output drivers or video filters to get the active
>   objects and help them to overlay them.

Yes. I think this should be handled separately from the "input API"
though; there's no need to change both at once.

> That's all for now: I will wait for some possible feedback before ranting
> any longer.

I think your comments are overall too vague and general to really
progress anywhere. You should identify some particular change you want
to make or issue you want to fix. Continuing discussion at this level
will generate talk but probably little else.