[MPlayer-dev-eng] [RFC] EOSD improvements

[Nicolas George]

Le quintidi 15 thermidor, an CCXVIII, Grigori Goronzy a écrit :
> What I am thinking of actually isn't that far off IMO, maybe we're
> simply misunderstanding each other? :)

I think the misunderstanding is mostly gone.

> a) pull all libass specifics/dependencies out of the EOSD, vo and vf_ass
> code and extract an indedependently usable interface while not breaking
> any current functionality.

I believe there is consensus about it being a good thing. I have a few ideas
about this point I would like to share with you and the rest of the list. I
will come back to them in a few paragraphs.

> b) extend EOSD with RGBA bitmap support and possibly more; what exactly
> depends on the possible use cases we can think of and what hardware is
> capable of.

This is Reimer's major point: this should wait later; at the very least
after your (d) point. And he convinced me: I am convinced that support for
colored bitmaps formats is necessary for a good OSD system, but it is not
immediately necessary to do (d). And the bonus: if (d) is done, it becomes
that much easier to test (b) immediately and get it working well.

> d) Modify spudec (and old-style OSD) to use the EOSD interface. IMHO
> spudec is a good fit to be a generic higher-level subpicture interface
> that deals with timing, caching, bitmap conversion, scaling, etc.

Where did (c) go? ;-)

I'm not sure about that. The code is currently quite complex and hardcodes
some features that would better be elsewhere. I think that starting with a
fresh eosd.c file, where only clean reviewed code goes, would be more
efficient.


As for my ideas about (a):

The data flow in current EOSD:

  When EOSD is performed by vf_ass, the steps are thus:

    1. Rendering settings are sent to libass during config.

    2. VFCTRL_DRAW_EOSD is a nop.

    3. EOSD is drawn in put_image.

    4. The bitmaps are obtained by calling ass_mp_render_frame in put_image.

  When EOSD is performed by the VO, the steps are thus:

    1. The drawing of EOSD is triggered by VFCTRL_DRAW_EOSD caught in vf_vo.

    2. Rendering settings are sent to libass for each frame.

    3. vf_vo gets the bitmaps (using ass_mp_render_frame) and sends them to
      the VO as the data for VOCTRL_DRAW_EOSD.

  I see several flaws in this design:

    1. EOSD has code specific to libass; this is easily corrected.

    2. EOSD can only show subtitles for one source. This is not a problem is
       the source is itself aggregating from several sources. (By source, I
       mean subtitles, timestamp in the corner, libmenu, etc.)

    3. The different structure between vf_ass and the VOs makes it much more
       difficult to understand what is going on.

    4. The difficult structure between vf_ass and the VOs makes it much more
       difficult to factorize similar code.

    5. I believe vf_vo should not hold any non-trivial EOSD-specific code,
       let alone ASS-specific; it should just forward the control messages.

       (In fact, I do not understand the reason for the distinction between
       VFCTRL and VOCTRL;°

Design of the classic OSD system:

  1. OSD can aggregate overlays from several sources, but the list of
     sources is hardcoded in several places of the code. There is no way for
     an option to add a new OSD element. For example, if I want to implement
     an option to show the DVB signal quality in the upper right corner,
     game over.

  2. Each OSD source is essentially a single rectangular overlay with a
     grayscale value and an alpha channel.

  3. OSD is drawn in reaction to the VFCTRL_DRAW_OSD, which vf_vo translates
     into a call to the draw_osd member function. For most VOs, draw_osd
     will call vo_draw_text with a "draw_alpha" callback function.
     vo_draw_text finds the OSD data in global variables.

  4. OSD support must be implemented in each VO. A lot of VOs ignore the
     draw_osd command or trigger a fatal error. Another lot of them just
     call vo_draw_text with a draw_alpha that blends the OSD into the
     incoming video. Only a handful of VOs actually do something smart.

  5. OSD makes a lot of effort to delay until the last moment the actual
     rasterization of some of the overlays. But not all. For example, the
     old SPU (DVD sub) decoder rasterizes at the last minute, while the new
     DVB subtitles decoder rasterizes in a completely different code path.

  6. For OSD elements that come from text, vo_ao uses directly the text to
     display it in ASCII.

  7. The parts of the code that modify OSD elements do so by acceding the
     global variables holding the values.

  Now, my remarks about that design:

  1. The allocation of OSD objects should be completely dynamic: any part of
     the code should be allowed to call some sort of "osd_add(...)",
     and later "osd_remove(...)".

  2. If we want to render the OSD objects through the EOSD renderers, we
     have either to emulate the value channel with two alpha channels (a
     completely white object on top of a completely white mask) or patch the
     EOSD renderers to accept a value channel. Both should be quite simple.
     The first is probably best to start with: the changes are more
     localized and easier to test. The second would come automatically if
     full color support is added to the EOSD renderers.

  4. If the EOSD renderers are used, the few OSD renderers that do something
     smart could be converted to EOSD.

  5. Currently, the late rasterization relies heavily on "switch(obj->type)"
     structures. This is not acceptable with regard to point (1). It could
     be achieved with the usual method for lazy evaluation in C: callbacks
     functions. But I think this is useless: just drop the late
     rasterization, and always enqueue bitmaps. That is what the DVB / XSUB
     / PGS renderer does.

  6. vo_aa is a toy: I do not think it matters very much if the support for
     its ASCII OSD breaks for some time. Adding it back should be pretty
     easy later.

Now, before I conclude that already too-long mail, what I think could be
done:

1. Create eosd.c for all common EOSD-related functions.

2. In vf_ass and vf_vo, replace calls to ass_mp_render_frame by calls to
   eosd_get_overlays; eosd_get_overlays itself would call
   ass_mp_render_frame. Idem with ass_configure and similar calls.

3. As an experimental patch, block VFCTRL_DRAW_OSD and change
   eosd_get_overlays to add the OSD objects to the ASS images.

If (3) works without too much effort, we know we are on the right track, and
we can start changing eosd_get_overlays to handle cleanly a dynamic set of
image sources.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/attachments/20100803/9d60c7fc/attachment-0001.pgp>