17 Oct
2018
17 Oct
'18
10:36 a.m.
I recently read this post [1] where it shows how to define a regular expression (using the
PCRE) syntax that looks very much like a proper grammar. A reduced example for the post
looks like this:
/
(?(DEFINE)
(?<addr_spec> (?&local_part) @ (?&domain) )
(?<local_part> (?&dot_atom) | (?"ed_string) |
(?&obs_local_part) )
(?<domain> (?&dot_atom) | (?&domain_literal) | (?&obs_domain) )
)
^(?&addr_spec)$
/x
The three capture groups “addr_spec”, “local_part” and “domain” would be the grammar
rules. It uses the (?&name) syntax to refer to another subgroup. TextMate does not
support that syntax but supports the following syntax: \g<name>, which the
documentation refers to as Subexp call [2]. This syntax seems to have the same semantics.
(DEFINE) is something that seems to be PCRE specific and basically means that the
following patterns will not be tried to match. It basically gives a place to define
subpatterns. I didn’t find anything corresponding in the TextMate regular expression
syntax but defining an optional group can be used as a workaround.
Here’s an example where I tried this technique to match a module declaration in the D
language:
(?:
(?<module_declaration>(?<module>module)\s+\g<module_fully_qualified_name>\s*;)
(?<module_fully_qualified_name>\g<module_name>|\g<packages>\.\g<module_name>)
(?<module_name>\g<identifier>)
(?<packages>\g<package_name>|\g<package_name>\.\g<packages>)
(?<package_name>\g<identifier>)
(?<identifier>\w+)
)?
\g<module_declaration>
This is exactly according to the specified grammar [3] and it seems to be working as
expected. Not sure if the optional group workaround causes some performance implications.
This technique seems like it could be a viable alternative to supporting variables in the
TextMate grammar as has been discussed before. What’s missing from this to make it really
useful would be something like (DEFINE) in PCRE and a place in the TextMate grammar to
place generic patterns used in multiple rules, like a pattern for identifiers.
[1] https://nikic.github.io/2012/06/15/The-true-power-of-regular-expressions.ht…
<https://nikic.github.io/2012/06/15/The-true-power-of-regular-expressions.html>
[2] https://macromates.com/manual/en/regular_expressions
<https://macromates.com/manual/en/regular_expressions>
[3] https://dlang.org/spec/grammar.html#ModuleDeclaration
--
/Jacob Carlborg
21 Oct
21 Oct
6:54 a.m.
On 17 Oct 2018, at 17:36, Jacob Carlborg wrote:
This is exactly according to the specified grammar [3]
and it seems to
be working as expected. Not sure if the optional group workaround
causes some performance implications.
This technique seems like it could be a viable alternative to
supporting variables in the TextMate grammar as has been discussed
before.
Just to be clear, you are talking about variables from the parsed
language and highlighting them later in the scope, right?
So something like: `let foo = 42 in … something_with foo …` and here
the latter instance of `foo` would be marked as a variable?
To be honest, I don’t follow this 100%; it seems highly impractical to
have one big regular expression for the language because then we cannot
save parser state and restart parser on the line that got edited, which
would be a major performance issue.
I am also not sure how this variable thing would work in a typical
language where you can define an arbitrary number of variables on
basically any given line.
3:32 p.m.
On 21 Oct 2018, at 08:54, Allan Odgaard
<mailinglist(a)textmate.org> wrote:
On 17 Oct 2018, at 17:36, Jacob Carlborg wrote:
This is exactly according to the specified grammar [3] and it seems to be working as
expected. Not sure if the optional group workaround causes some performance implications.
This technique seems like it could be a viable alternative to supporting variables in the
TextMate grammar as has been discussed before.
Just to be clear, you are talking about variables from the parsed language and
highlighting them later in the scope, right?
So something like: let foo = 42 in … something_with foo … and here the latter instance of
foo would be marked as a variable?
No, I’m talking about having variables in the grammar with reusable regular
expression snippets. Something like:
{
patterns = (
{
name = 'meta.definition.class.d';
match = 'class $identifier’; // the “identifier” variable
referenced/interpolated
}
);
repository = {
variables = {
identifier = '(\w+)';
};
};
}
In the above example “identifier” would be a variable that can be interpolated in other
rules.
The above example would be one approach. Another approach to support variables is what I
showed in my original post but using already existing features of regular expression.
{
patterns = (
{
name = 'meta.definition.class.d';
match = ‘\g<class_declaration>’; // this is standard regular expression syntax
that works today in TextMate we just need somewhere to define “class_declaration"
},
{
name = ‘meta.definition.struct.d’;
match = ’struct\s+\g<identifier>’; // reusing the “identifier” sub expression
in another pattern
}
);
repository = {
defines = '(?x) // here sub expressions can be defined which can later be used in
“patterns"
(?<class_declaration>class\s+\g<identifier>)
(?<identifier>\w+)
‘;
};
}
This is all about how one can write/implement a TextMate grammar. This approach would
allow to write a TextMate grammar that closely follows a “real” grammar for a programming
language, like a BNF grammar. I hope this makes it a bit more clear.
--
/Jacob Carlborg
1 Nov
1 Nov
6:35 a.m.
On 21 Oct 2018, at 22:32, Jacob Carlborg wrote:
No, I’m talking about having variables in the grammar
with reusable
regular expression snippets. Something like:
[…]
I see, well for that, I think it would be more consistent (with the
existing feature set) to treat regular expressions as format strings, as
we already use format strings in other places related to grammars, and
as variables can be referenced is `${…}` it should not really cause an
issue with escaping, as the only thing that should meaningfully go after
`$` in a regexp is `\n` (or possibly '(').
899
days inactive
914
days old
3 comments
2 participants
participants (2)
-
Allan Odgaard -
Jacob Carlborg