Hi, the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8. Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future. Greetings, Reimar Döffinger
Hi, On Sunday 23 April 2006 06:09, Reimar Döffinger wrote:
the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8.
Tested under a UTF-8 locale, works ok.
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
libmenu could use the same kind of code if it want to support localized menu item IMHO. -- Best Regards, LR
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Hi, the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8. Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace. Greetings, Reimar Döffinger
On Tue, Apr 25, 2006 at 02:56:59PM +0200, Reimar D�ffinger wrote:
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace.
What about storing strings internally in UCS-2? Parse once when reading so there will be no need to recode UTF-8 -> UCS-2 before displaying. Regards -- Adam Tlałka mailto:atlka@pg.gda.pl ^v^ ^v^ ^v^ PGP public key: finger atlka@sunrise.pg.gda.pl
On Friday, 28 April 2006 at 00:56, Adam Tlałka wrote:
On Tue, Apr 25, 2006 at 02:56:59PM +0200, Reimar D�ffinger wrote:
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace.
What about storing strings internally in UCS-2? Parse once when reading so there will be no need to recode UTF-8 -> UCS-2 before displaying.
We're not going to use UTF-16 internally. R. -- MPlayer developer and RPMs maintainer: http://rpm.greysector.net/mplayer/ There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan
2006/4/28, Adam Tlałka <atlka@pg.gda.pl>:
On Tue, Apr 25, 2006 at 02:56:59PM +0200, Reimar D�ffinger wrote:
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace.
What about storing strings internally in UCS-2? Parse once when reading so there will be no need to recode UTF-8 -> UCS-2 before displaying.
If I remember right UCS-2 is 2 byte unicode. Unfortunately the Unicode standard wasn't finished at the time M$ created that. It was finally decided that full unicode is 4 bytes (Of course not the whole range is used). This requiring some hacks to be made to support UCS-2 (this is what Rathann is referring when pointing the UTF-16). Using fixed size characters may have some benefits, but in our case we don't need it.
On Fri, Apr 28, 2006 at 10:46:10AM +0300, Ivan Kalvachev wrote:
2006/4/28, Adam Tlałka <atlka@pg.gda.pl>:
On Tue, Apr 25, 2006 at 02:56:59PM +0200, Reimar D�ffinger wrote:
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace.
What about storing strings internally in UCS-2? Parse once when reading so there will be no need to recode UTF-8 -> UCS-2 before displaying.
If I remember right UCS-2 is 2 byte unicode. Unfortunately the Unicode standard wasn't finished at the time M$ created that. It was finally decided that full unicode is 4 bytes (Of course not the whole range is used). This requiring some hacks to be made to support UCS-2 (this is what Rathann is referring when pointing the UTF-16).
Using fixed size characters may have some benefits, but in our case we don't need it.
In libvo/sub.c -> vo_update_text_sub in case of UTF-8 subs we recode UTF-8 to font positions - 2 and 3 bytes sequences which means U+00080..U+00FFFF so it is just an UCS-2 Unicode representation.;-) Of course this UTF-8 -> UCS-2 (2 byte Unicode) decoder is quite simple and not detecting inproper UTF-8 seqences. But its fast. If iconv is working properly then there should be no problem with that approach but in case of broken UTF-8 encoded subtitles it can lead to undesired effects - we just read the text from file - no UTF-8 check. So maybe better is to encode directly to UCS-2 or just to the font positions while reading subtitle files and detect errors there and not in the subtitle display fuction? Anyway in case where sub_utf8 is not set we already treat vo_sub->text bytes as font glyphs positions so maybe it always can have this meaning and we can treat them as u8 or u16 glyphs positions depending on some flag (sub_ucs 0:u8 1:u16)? It is more universal and could be prepared even for UCS-4 in the future. Regards -- Adam Tlałka mailto:atlka@pg.gda.pl ^v^ ^v^ ^v^ PGP public key: finger atlka@sunrise.pg.gda.pl
On Fri, Apr 28, 2006 at 12:56:49AM +0200, Adam Tla?ka wrote:
On Tue, Apr 25, 2006 at 02:56:59PM +0200, Reimar D???ffinger wrote:
Hi, On Tue, Apr 25, 2006 at 09:54:02AM +0300, Ivan Kalvachev wrote:
2006/4/23, Reimar Döffinger <Reimar.Doeffinger@stud.uni-karlsruhe.de>:
Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
I think this is going to be the second utf-8 parser in that file, as subs are (usually) already utf-8. Take a look of vo_update_text_sub::382
I know, but that code could in case of invalid encoding skip the terminating NULL and cause a segfault. Thus it is unsuitable for most cases, and this is actually one of the utf8-parsers I'd like to replace.
What about storing strings internally in UCS-2?
ABSOLUTELY NOT! UCS-2 is an abomination and should never be used for anything. Maybe you meant UCS-4? Still I'm against it. It should be possible to process strings as strings EVERYWHERE up until the point where the actual characters are printed. This is not possible with any encoding but UTF-8. Rich
Hi, On Sunday 23 April 2006 06:09, Reimar Döffinger wrote:
Hi, the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8. Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
Is there a conclusion made about this patch, will it be applied before the pre8 release? -- Best Regards, LR
2006/5/12, 陆然 <hephooey@gmail.com>:
Hi,
On Sunday 23 April 2006 06:09, Reimar Döffinger wrote:
Hi, the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8. Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
Is there a conclusion made about this patch, will it be applied before the pre8 release?
We are in bug-fix only stage.
Hi, On Friday 12 May 2006 17:19, Ivan Kalvachev wrote:
We are in bug-fix only stage.
Well, then I think we should disable the osd localization code, any non-latin1 transaltion will give garbage in OSD, that will make users think there is a bug instead of a missing feature :( -- Best Regards, LR
2006/5/12, 陆然 <hephooey@gmail.com>:
Hi,
On Friday 12 May 2006 17:19, Ivan Kalvachev wrote:
We are in bug-fix only stage.
Well, then I think we should disable the osd localization code, any non-latin1 transaltion will give garbage in OSD, that will make users think there is a bug instead of a missing feature :(
It may be better to commit the code then.... Well, it's up to Reimar.
On Fri, May 12, 2006 at 01:02:40PM +0300, Ivan Kalvachev wrote:
2006/5/12, ???? <hephooey@gmail.com>:
Hi,
On Friday 12 May 2006 17:19, Ivan Kalvachev wrote:
We are in bug-fix only stage.
Well, then I think we should disable the osd localization code, any non-latin1 transaltion will give garbage in OSD, that will make users think there is a bug instead of a missing feature :(
It may be better to commit the code then.... Well, it's up to Reimar.
I consider this code to be bugfix code.. Rich
Hi, On Fri, May 12, 2006 at 11:24:36AM -0400, Rich Felker wrote:
It may be better to commit the code then.... Well, it's up to Reimar.
I consider this code to be bugfix code..
It might break when somebody compiles/configures with --charset=noconv or so, then our localizations are not in UTF8. Probably same for the bitmap fonts (at least I doubt they were all fixed to work with unicode). IOW, it has some shortcomings, I won't apply unless someone urges me to do it (or apply themselves). Somewhat related: I'd really prefer to make all the localized files UTF8 instead of the current default, getting rid of the need for an iconv run for most compiles. Greetings, Reimar Döffinger
On Fri, May 12, 2006 at 07:04:03PM +0200, Reimar Döffinger wrote:
Hi, On Fri, May 12, 2006 at 11:24:36AM -0400, Rich Felker wrote:
It may be better to commit the code then.... Well, it's up to Reimar.
I consider this code to be bugfix code..
It might break when somebody compiles/configures with --charset=noconv or so, then our localizations are not in UTF8. Probably same for the bitmap fonts (at least I doubt they were all fixed to work with unicode). IOW, it has some shortcomings, I won't apply unless someone urges me to do it (or apply themselves). Somewhat related: I'd really prefer to make all the localized files UTF8 instead of the current default, getting rid of the need for an iconv run for most compiles.
You have to use iconv in case of loading subtitle files or you must have them already recoded to UTF-8. So iconv(3) will be used anyway or you must force users to recode their subtitle files to get rid of iconv from mplayer's code. So think carefully ;-). Localized messages are not so bad but in case of non UTF-8 encoding they simply should have some tag which indicates which encoding is used inside. It is not a big problem IMHO. Regards -- Adam Tlałka mailto:atlka@pg.gda.pl ^v^ ^v^ ^v^ PGP public key: finger atlka@sunrise.pg.gda.pl
Hi, On Fri, May 12, 2006 at 08:30:03PM +0200, Adam Tla?ka wrote:
Somewhat related: I'd really prefer to make all the localized files UTF8 instead of the current default, getting rid of the need for an iconv run for most compiles.
You have to use iconv in case of loading subtitle files or you must have them already recoded to UTF-8. So iconv(3) will be used anyway or you
I am talking about iconv(P) during make. Greetings, Reimar Döffinger
On Fri, May 12, 2006 at 11:53:33PM +0200, Reimar Döffinger wrote:
Hi, On Fri, May 12, 2006 at 08:30:03PM +0200, Adam Tla?ka wrote:
Somewhat related: I'd really prefer to make all the localized files UTF8 instead of the current default, getting rid of the need for an iconv run for most compiles.
You have to use iconv in case of loading subtitle files or you must have them already recoded to UTF-8. So iconv(3) will be used anyway or you
I am talking about iconv(P) during make.
Hmm, OK my misunderstanding. So do we really need to use the iconv program? Regards -- Adam Tlałka mailto:atlka@pg.gda.pl ^v^ ^v^ ^v^ PGP public key: finger atlka@sunrise.pg.gda.pl
Hi, On Saturday 13 May 2006 01:04, Reimar Döffinger wrote:
It might break when somebody compiles/configures with --charset=noconv or so, then our localizations are not in UTF8. Probably same for the bitmap fonts (at least I doubt they were all fixed to work with unicode). IOW, it has some shortcomings, I won't apply unless someone urges me to do it (or apply themselves). Somewhat related: I'd really prefer to make all the localized files UTF8 instead of the current default, getting rid of the need for an iconv run for most compiles.
So what are we going to do with pre8? 1) Apply the patch. 2) Do not localize the OSD message. 3) Make all the help files UTF-8. 4) Leave it as it is now. I think 4) is unacceptable, 3) may be the best solution, but it might introduce some encoding bugs during the conversion and delay the release. If you feel so disgusting to apply the patch, I think the only choice is 2). Any better ideas? -- Best Regards, LR
Hi, On Sat, May 13, 2006 at 10:16:18AM +0800, ?????? wrote:
So what are we going to do with pre8? 1) Apply the patch. 2) Do not localize the OSD message. 3) Make all the help files UTF-8. 4) Leave it as it is now. I think 4) is unacceptable, 3) may be the best solution, but it might introduce some encoding bugs during the conversion and delay the release. If you feel so disgusting to apply the patch, I think the only choice is 2). Any better ideas?
Well, the only point is that this is a patch for a feature I don't use and I'm not sure if it is an improvement. So, suggestion: If two people on this list tell me they tested this patch, it is a significant improvement for them and they want it in pre8, and there are no objections, I'll apply it. Greetings, Reimar Döffinger
Hi, On Saturday 13 May 2006 20:05, Reimar Döffinger wrote:
Well, the only point is that this is a patch for a feature I don't use and I'm not sure if it is an improvement. So, suggestion: If two people on this list tell me they tested this patch, it is a significant improvement for them and they want it in pre8, and there are no objections, I'll apply it.
OK, I've tested the patch, it's ok on a Gentoo Box with zh_CN translation, the locale settting is zh_CN.utf8. Check out the two screenshots in the attachment to see the significant improvement ;) -- Best Regards, LR
Hi, On Sat, May 13, 2006 at 09:53:51PM +0800, ?????? wrote:
OK, I've tested the patch, it's ok on a Gentoo Box with zh_CN translation, the locale settting is zh_CN.utf8. Check out the two screenshots in the attachment to see the significant improvement ;)
So, is there a second tester/proponent of this patch to be found? ;-) Greetings, Reimar Döffinger
Hi, On Sun, Apr 23, 2006 at 12:09:10AM +0200, Reimar D?ffinger wrote:
the attached patch adds support for OSD messages in UTF-8. Though this probably should only be active when MSG_CHARSET actually is UTF-8. Also, the utf8_get_char function is a bit overkill for this, since it will always encouter valid UTF-8 in this case, but I made it more flexible in the hope in will be used in other place as well in the future.
Release is over, so applied. Greetings, Reimar Döffinger
participants (7)
-
Adam Tlałka -
Dominik 'Rathann' Mierzejewski -
Ivan Kalvachev -
Reimar Döffinger -
Reimar D�ffinger
-
Rich Felker -
陆然