On 25/07/2005, at 2.14, Sean Schertell wrote:

Unfortunately, UTF-8 doesn't work for Japanese in some cases. For example Mac IE (still used by all those poor souls still stuck on OS 9), shows lots of wacky characters.

Sorry to sound skeptical, but is that just a subset of the japanese characters, which it shows as wacky?

I tried launching Internet Explorer 5.2 (which is on my Panther partition) and went to http://en.wikipedia.org/wiki/Japanese_language -- this page is in UTF-8 and contain several japanese characters, and although I don't read japanese, it does look alright to me.

Remember, Japan is a huge (possibly #2) market for the web so this isn't some obscure out in left field feature ;-)

I'm going to add better/actual international support to TM eventually. Though currently it looks a little like it'll be a 1.4- thing (but 1.2 and 1.3 should take shorter time than the current 1.1).

I personally would doubt it would happen either, considering the fact that UTF-8 handles japanese input, and the reason unicode exists is to make other character encodings obsolete in favor of one encoding that handles everything.

On 7/25/05, Sune Foldager cryo@cyanite.org wrote:

On 25/07/2005, at 14.19, Sean Schertell wrote:

...

This is my first post. I just wanna say: Wow! This thing looks truly awesome! But... for those of us doing non-western and non- UTF-8 encoding web pages, it's unusable. Oh please please PLEASE add support for Shift-JIS and EUC-JP! (japanese)

I am almost positive it won't happen; you can use external tools such as iconv or similar to convert to/from instead. The best would be to use UTF-8 on those web pages of course.

-- Sune.

---

For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate

...

I am almost positive it won't happen; you can use external tools such as iconv or similar to convert to/from instead. The best would be to use UTF-8 on those web pages of course.

UTF-8 sucks for Japanese and Chinese texts mainly due to space reasons. If anything makes sense, then it is UTF-16, which Textmate also supports.

Apart from that, it seems that Unicode is not actually able to handle 100% of Chinese (and maybe also Japanese) script. But I'm not a Unicode expert.

Could you explain what you mean by "space reasons"?

Also, does anyone else have any experience using UTF-8 for Japanese web pages? It seems that none of the major Japanese sites use it (they mostly use Shift-JIS). I tried to use it once but discovered that Mac IE 5 rendered form elements such as pull down lists and buttons as "mojibake" (garbage characters). Maybe there's a workaround for that?

If anyone has success stories with Japanese UTF-8, I'd love to hear them. I certainly don't have any personal qualms about leaving Shift- JIS behind -- as long as I know UTF-8 is viable for commercial Japanese web sites.

Sean

:::: DataFly.Net :::: Complete Web Services http://www.datafly.net

On 26/07/2005, at 6.26, Patrice Neff wrote:

[...] while with English and most European languages you will save a lot of space using UTF-8 compared to UTF-16. And the latter was IMHO one of the main reasons for developing UTF-8.

Well, at best you'll save 50%, where enabling gzip as transfer- compression will likely save you >75% :)

The motivation for UTF-8 is that ASCII characters are encoded as they would have been, had it been a plain ASCII document.

This means that a lot of existing software doesn't need to be updated to actually handle UTF-8 (as long as they are 8 bit clean). For example I use UTF-8 for my source code, even though my compiler isn't UTF-8 aware, this means I can use non-ASCII in strings and comments -- some compilers/interpreters (e.g. PHP) will also allow user defined variables to be in UTF-8 (while still only knowing about the ASCII tokens).

So UTF-8 exists because a lot of software is made to work with 8-bit sequences (not 16 bit, as UTF-16 would have called for), and some software will look for tokens encoded as ASCII in these 8-bit sequences.

UTF-8 is a brilliant way to give this software access to the full unicode range.

Hi,

...

This means that a lot of existing software doesn't need to be updated to actually handle UTF-8 (as long as they are 8 bit clean). For example I use UTF-8 for my source code, even though my compiler isn't UTF-8 aware

Just a thought; what happens when/if a unicode character is coded as <something><the code for "> in a string? I suppose C will fail in that case?

It's not possible. When you get a byte part of an UTF-8 sequence, you can know if its position in the sequence by checking the value of a few bits. An ASCII character (its value is less than 127) can only be alone.

Cheers, Vincent ISAMBART