[TxMt] New syntax engine

[Allan Odgaard]

On Mar 16, 2005, at 0:59, David Lee wrote:

Please compose a new letter when starting a new thread, for more info 
see item 2 here: http://linux.sgms-centre.com/misc/netiquette.php

> I heard somewhere the next release will overhaul the syntax 
> highlighting
> engine.
>
> Is there a description of the changes / new approach anywhere ?

Only loose comments made by me, so to sum it up:

In parsing:
  Full recursion is supported, so e.g. interpolated code in strings work 
etc.
  Back-references in end pattern, so e.g. heredocs are possible (at 
least when they are not nested)
  Zero-width matches supported, rarely needed, but required for some 
stuff
  The \n at the end of a line can be included in a match, this is mostly 
useful for styling (to make a background color change extend to the 
right border)
  Captures can be named (this is important for the next part)

In styling:
  Styles are associated to a match using the name/prefix of the rule, 
optionally preceding it with the name/prefix of parent language rules 
for context, example:
   strings               -- match all strings
   strings.double-quoted -- match all double-quoted strings
   source.php strings    -- match only strings with a parent element 
named source.php (i.e. strings in php sources)

Basically this is the same as you do with CSS selectors. But I will 
allow for more than just visual styles, e.g. disable ' as a smart 
typing character in text.html but enable it again for markup.tags, or 
disable continuous spell checking for source but enable it again for 
source strings, source comments etc.

This same system is also what you will be able to fill in to the scope 
string in the bundle editor. That means that only in that scope, will 
the activation method be functional. So the most obvious is to limit 
snippets to their respective source, but we can also do much fancier 
stuff such as overload the behavior of return, but limited to e.g. 
strings -- this has great potential.

But all in all, the old syntax files are compatible -- there'll just be 
slightly more flexibility in the definition, and I'll ignore the styles 
embedded in these files, since that will instead be applied using style 
sheets -- but it should be relatively easy to extract the styles from a 
given syntax file and create the corresponding style sheet.