Hans-Joerg Bibiko wrote:

Dear all,

there's a new bundle called "Unicode" in the review trunk. It is meant to be a place where we can gather any kind of scripts, commands, etc. which are related to general Unicode issue, meaning non-ASCII. This should also a place where we can gather scripts related to specific languages like Japanese, Chinese, Greek etc. This bundle is the first stage. How do we separate this bundle is a future task.

Thus, if there is someone who already has such scripts or is willing to support, please let us/me know.

Up to now there are the following stuff in:

• Normalize according canonical (de)composition of accented characters
• Delete Diacritics: façadë έ だ => facade ε た
• Convert to a similar Unicode Character: type the letter 'c' to get a

list of "cçćĉċčƈ¢ɕʗḉ⒞ⓒｃ￠"

• Convert to Greek Character: type 'n' to get "ν"
• Show Unicode Name: select some letters to get a list of the Unicode

names like LATIN SMALL LETTER A

I have many other scripts, but I need some time to polish them up.

To get this bundle, simply use the Subversion Bundle's checkout

http://macromates.com/svn/Bundles/trunk/Review/Bundles/Unicode.tmbundle

save this to the Desktop or whatever.

I know, to deal with non-ASCII scripts in TM 1.x is a bit tricky, but TM 2.0 will come ;)

One small note:

In the character name script you should probably call unicodedata.name() with a second argument in case the character has no name, i.e. replace

res = a + " : " + unicodedata.name(a)

with

res = a + " : " + unicodedata.name(a, "U+%04X" % ord(a))

Furthermore it would be great if this script could display all information there is in the Python Unicode database, i.e. stuff like

unicodedata.category() unicodedata.bidrectional() unicodedata.decimal()

etc.

Servus, Walter

Sorry but do i miss something? I have that error ------------------------- Traceback (most recent call last): File "/tmp/temp_textmate.0WYiu4", line 50, in <module> result=dialog.menu([re.sub(r"(?=[^a-zA-Z0-9_ ./-\x7F-\xFF\n])", r'\', a) + "\t" + unicodedata.name(a, "U+%04X" % ord(a)) for a in suggestions]) File "/Applications/TextMate.app/Contents/SharedSupport/Support/lib/dialog.py", line 51, in menu plist = to_plist(menu) UnboundLocalError: local variable 'menu' referenced before assignment ------------------------- when try to "Convert to Greek..." or "Convert to Similar..."

Alexey Blinov

On Mon, Jun 2, 2008 at 3:09 AM, Hans-Jörg Bibiko bibiko@eva.mpg.de wrote:

On 02.06.2008, at 00:04, Walter Dörwald wrote:

...

Here's another patch (against the current version). It shows both the codepoint and the name.

BTW, you don't have to use a regular expression to split a string into characters, simply iterating through it does the trick:

Index: Commands/Show Unicode Names.tmCommand -for a in re.compile("(?um)(.)").split(unicode(sys.stdin.read(), "UTF-8")):

• if (len(a)==1) and (a != '\n'):
  

•      res = a + " : " + unicodedata.name(a, "U+%04X" % ord(a))
  

+for a in unicode(sys.stdin.read(), "UTF-8"):

• if a != '\n':
  

•      res = u"%s : U+%04X" % (a, ord(a))
  

•      name = unicodedata.name(a, None)
  

•      if name:
  

•          res += u" : %s" % name
      print res.encode("UTF-8")</string>
   <key>fallbackInput</key>
   <string>character</string>
  

Thanks! Just committed to the trunk.

...

...

...

Furthermore it would be great if this script could display all information there is in the Python Unicode database, i.e. stuff like

unicodedata.category() unicodedata.bidrectional() unicodedata.decimal()

Yes. I have such a script in Perl which also shows up info about Unicode code points etc.

Just added to the bundle a prototype of 'Show Unicode Properties'

...

Another problem: Using Ctrl-Shift-U as the shortcut hides the "Convert To Lowercase" command.

Yes. This was a bad key combo. I changed it temporally to CTRL+OPT+APPLE+U

BTW: Can Python handle Unicode codepoints which are specified in Unicode pane B, meaning greater U+FFFF? I tried it out. I found out that Python uses UTF-16 internally. But e.g. UCS hex: 20000 ; UTF-16: D840 DC00 . I can print that character to TM but unicodedata fails because it expects one character but not two (?)

Servus,

--der Hans ______________________________________________________________________ For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate

Hans-Jörg Bibiko wrote:

On 02.06.2008, at 00:04, Walter Dörwald wrote:

...

Here's another patch (against the current version). It shows both the codepoint and the name.

BTW, you don't have to use a regular expression to split a string into characters, simply iterating through it does the trick:

Index: Commands/Show Unicode Names.tmCommand -for a in re.compile("(?um)(.)").split(unicode(sys.stdin.read(), "UTF-8")):

• if (len(a)==1) and (a != '\n'):
  

•      res = a + " : " + unicodedata.name(a, "U+%04X" % ord(a))
  

+for a in unicode(sys.stdin.read(), "UTF-8"):

• if a != '\n':
  

•      res = u"%s : U+%04X" % (a, ord(a))
  

•      name = unicodedata.name(a, None)
  

•      if name:
  

•          res += u" : %s" % name
       print res.encode("UTF-8")</string>
  

  <key>fallbackInput</key> <string>character</string>

Thanks! Just committed to the trunk.

...

...

...

Furthermore it would be great if this script could display all information there is in the Python Unicode database, i.e. stuff like

unicodedata.category() unicodedata.bidrectional() unicodedata.decimal()

Yes. I have such a script in Perl which also shows up info about

Unicode

...

code points etc.

Just added to the bundle a prototype of 'Show Unicode Properties'

...

Another problem: Using Ctrl-Shift-U as the shortcut hides the "Convert To Lowercase" command.

Yes. This was a bad key combo. I changed it temporally to CTRL+OPT+APPLE+U

BTW: Can Python handle Unicode codepoints which are specified in Unicode pane B, meaning greater U+FFFF? I tried it out. I found out that Python uses UTF-16 internally.

At least the Python that ships with the OS uses 2 byte Unicode character with partial UTF-16 support:

Python 2.5.2 (r252:60911, Apr 8 2008, 18:54:00) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

import sys sys.maxunicode

65535

The size of a Unicode character is specified at compile time with the --enable-unicode option, so you *could* compile a wide Python with: ./configure --enable-unicode=ucs4

But e.g. UCS hex: 20000 ; UTF-16: D840 DC00 . I can print that character to TM but unicodedata fails because it expects one character but not two (?)

There are some spots in the Python code base where in narrow builds surrogate pairs are interpreted properly as characters outside the BMP, but unicodedata isn't one of them (so it's not actually real UTF-16 throughout). There's an open issue on the Python bugtracker about that:

http://bugs.python.org/issue1706460

So there are two options:

1) Apple starts compiling its Python with --enable-unicode=ucs4 2) Python gets fixed so that surrogate pairs can be passed to unicodedata functions.

I think I might give 2) a try.

Servus, Walter

Walter Dörwald wrote:

Walter Dörwald wrote:

...

Hans-Jörg Bibiko wrote:

...

On 02.06.2008, at 00:04, Walter Dörwald wrote:

...

Here's another patch (against the current version). It shows both the codepoint and the name. [...]

Here's another suggestions on the current Bundle version:

To get the UTF-8 bytes of a character, you're doing the following:

print "  UTF-8         : " + " 

".join(repr(char.encode("UTF-8")).split('\x')).lstrip("' ").rstrip("'").upper()

This only works for characters with a codepoint >= 128. The following code should work better:

print "  UTF-8         : %s" % " ".join(hex(ord(c))[2:].upper() for 

c in char)

Oops, that was of course supposed to be:

print " UTF-8 : %s" % " ".join(hex(ord(c))[2:].upper() for c in char.encode("utf-8"))

Servus, Walter

On 03.06.2008, at 17:29, Walter Dörwald wrote:

Walter Dörwald wrote:

...

Walter Dörwald wrote:

...

Hans-Jörg Bibiko wrote:

...

On 02.06.2008, at 00:04, Walter Dörwald wrote:

...

Here's another patch (against the current version). It shows both the codepoint and the name. [...]

Here's another suggestions on the current Bundle version: To get the UTF-8 bytes of a character, you're doing the following: print " UTF-8 : " + " ".join(repr(char.encode ("UTF-8")).split('\x')).lstrip("' ").rstrip("'").upper() This only works for characters with a codepoint >= 128. The following code should work better: print " UTF-8 : %s" % " ".join(hex(ord(c))[2:].upper () for c in char)

Oops, that was of course supposed to be:

 print "  UTF-8         : %s" % " ".join(hex(ord(c))[2:].upper 

() for c in char.encode("utf-8"))

Could it be that this isn't allowed in Python for Tiger? I get an error for invalid syntax referring to 'for' Mac OSX 10.4.11 ppc; Python 2.4.2

On my 10.5.3 Mac it works(?)

--Hans

On 03.06.2008, at 22:44, Walter Dörwald wrote:

print " UTF-8 : %s" % " ".join([hex(ord(c))[2:].upper() for c in char.encode("utf-8")])

Thanks. This did the trick. I thought that I did try out this [ ]-notation, but anyway ... the main thing is that it works :)

--Hans