I've noticed that right now displaymath modes in latex initiated by $ $ are not handled correctly. It seems that lots of times, though not entirely consistently, they are being captured by the string.other.math.tex scope, instead of the string.other.math.block.latex scope. In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$ would consider the first pair of dollar signs as string.other.math.tex scope, and the second as the beginning of string.other.math.block.latex, which then goes on to capture the entire rest of the document. So I am wondering, how is this possible that the $$ is not consistently matched, and what can we do about it?
Haris
Haris,
How its possible is easy to explain... I don't ever use $$ to start a math mode, in fact until I saw this email I either did not know or had forgotten that $$ was even valid. So, for all the work I've done modifying both TeX and LaTeX modes I've never thought about it. Right now, I look to the test.tex file as the spec for this bundle. That is does everything in the test document look good, and is it scoped correctly.
I only know that parts of TeX and LaTeX that I use. I've never taken a step back and thought about how to structure the language definition from the top down. I'm not even certain that TeX is 'regular' enough to do that well.
I'm certainly open to suggestions from other TeX experts on how to make the bundle better or if there is a way to refactor what is already there to make it work more consistently. In the meantime, I'll add your example below to the test.tex file in the bundle and give it some thought.
Brad
On Nov 6, 2005, at 3:54 PM, Charilaos Skiadas wrote:
I've noticed that right now displaymath modes in latex initiated by $$ are not handled correctly. It seems that lots of times, though not entirely consistently, they are being captured by the string.other.math.tex scope, instead of the string.other.math.block.latex scope. In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$ would consider the first pair of dollar signs as string.other.math.tex scope, and the second as the beginning of string.other.math.block.latex, which then goes on to capture the entire rest of the document. So I am wondering, how is this possible that the $$ is not consistently matched, and what can we do about it?
Haris
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
On Nov 6, 2005, at 3:54 PM, Charilaos Skiadas wrote:
I've noticed that right now displaymath modes in latex initiated by $$ are not handled correctly. It seems that lots of times, though not entirely consistently, they are being captured by the string.other.math.tex scope, instead of the string.other.math.block.latex scope. In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$ would consider the first pair of dollar signs as string.other.math.tex scope, and the second as the beginning of string.other.math.block.latex, which then goes on to capture the entire rest of the document. So I am wondering, how is this possible that the $$ is not consistently matched, and what can we do about it?
I pasted your example above into the test.tex document and the text between the $$ .. $$ is scoped as string.other.math.tex The word 'where' is scoped as text.latex and the g,f is scoped as string.other.math.tex I believe those are all correct yes?
So, now I'm curious as to whats different between your document and the test.tex document.
Is there a a reason to make the block math scope different than the regular math scope? It seems like it might be better to move the $$ into the TeX syntax file so that the contents of a block math scope got the same treatment as an inline math math.tex scope?
Brad
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
Hi Brad, On Nov 6, 2005, at 4:19 PM, Brad Miller wrote:
On Nov 6, 2005, at 3:54 PM, Charilaos Skiadas wrote:
In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$
I pasted your example above into the test.tex document and the text between the $$ .. $$ is scoped as string.other.math.tex The word 'where' is scoped as text.latex and the g,f is scoped as string.other.math.tex I believe those are all correct yes?
So, now I'm curious as to whats different between your document and the test.tex document.
So in my case, if I put the cursor between the double dollar signs at the beginning, the scope is string.other.math.tex. If I put it anywhere between that set of double dollar signs and the next one (supposed to be closing the math mode, it is not. This is what's wrong in my case. If I put it between the second set of double dollar signs, it is string.other.math.block.tex.
I'll send you separately my file, see if that helps, but there's not anything weird about it I think.
Is there a a reason to make the block math scope different than the regular math scope? It seems like it might be better to move the $ $ into the TeX syntax file so that the contents of a block math scope got the same treatment as an inline math math.tex scope?
If it is allowed in TeX, and I think it is, then yes it should probably be there. I think historically the difference was, IIRC, that the regular math mode was matched with a "match" regex, while the displaymath with a "begin"-"end" pair, hence they were different. I do believe they should be dealt with the same way if possible. What happens in this case is that the first double dollar signs are matched with the regex from the TeX bundle: { name = 'string.other.math.tex'; begin = '$'; end = '$'; swallow = '\$'; ...
On the other hand, the second double dollar signs are matched as the begin regex from the LaTeX bundle: { name = 'string.other.math.block.latex'; begin = '$$'; end = '$$'; },
and the parser goes on to search the entire rest of the document for the closing $$ So what confuses me (and this is what I meant when I asked how it was possible) is how only the first of these is matched by the string.other.math.tex regex, and not the second one as well. How can string.other.math.tex not take priority in both cases? Maybe Allan can answer that, but for right now I see the following possible solutions: 1) Make it so that the string.other.math.tex regex needs to match something between the begin = '$'; end = '$'; Not sure if this is possible. 2) Move the string.other.math.block.latex in the TeX file, and place it above the string.other.math.tex pattern, so that it matches first. 3) Even better, mix the two patterns into one.
Brad
Haris
PS: Out of curiosity, what do you use instead of $$, and how did you historically come to do it that way? In the university of Chicago, I think we all do the displaymath mode that way. In fact, I haven't before met anyone not using it. Are there reasons against using it?
Haris,
Thanks for the file. I'll take a look at it and see if I can see something.
Rather than $$ I've always just used \begin{equation} or equation* if I didn't want equation numbers. Thats just how I learned it a long time ago, and I've never had cause to do anything different. From the LaTeX help site:
There are four environments that put LaTeX in math mode: math, displaymath, eqnarray, and equation. The math environment is for formulas that appear right in the text. The displaymath and equation environments are for formulas that appear on their own line; they differ only insofar as the latter prints an equation number. LaTeX will not break lines in displaymath or equation environments unless told to do so with a \ command. The displaymath environment can be entered by \begin{displaymath} math equation \ second line of math equation \end{displaymath} Because displaymath may be used frequently to display relatively short equations, there are convenient alternative forms: [ displayed math equation ] or (the following does not seem to be well documented, but appears to work) $$ dispalyed math equation $$
I don't think there is any difference between equation* and displaymath but I could be wrong...
I think for purposes of scoping in TextMate we ought to treat them all the same.
Brad
On Nov 6, 2005, at 5:09 PM, Charilaos Skiadas wrote:
Hi Brad, On Nov 6, 2005, at 4:19 PM, Brad Miller wrote:
On Nov 6, 2005, at 3:54 PM, Charilaos Skiadas wrote:
In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$
I pasted your example above into the test.tex document and the text between the $$ .. $$ is scoped as string.other.math.tex The word 'where' is scoped as text.latex and the g,f is scoped as string.other.math.tex I believe those are all correct yes?
So, now I'm curious as to whats different between your document and the test.tex document.
So in my case, if I put the cursor between the double dollar signs at the beginning, the scope is string.other.math.tex. If I put it anywhere between that set of double dollar signs and the next one (supposed to be closing the math mode, it is not. This is what's wrong in my case. If I put it between the second set of double dollar signs, it is string.other.math.block.tex.
I'll send you separately my file, see if that helps, but there's not anything weird about it I think.
Is there a a reason to make the block math scope different than the regular math scope? It seems like it might be better to move the $$ into the TeX syntax file so that the contents of a block math scope got the same treatment as an inline math math.tex scope?
If it is allowed in TeX, and I think it is, then yes it should probably be there. I think historically the difference was, IIRC, that the regular math mode was matched with a "match" regex, while the displaymath with a "begin"-"end" pair, hence they were different. I do believe they should be dealt with the same way if possible. What happens in this case is that the first double dollar signs are matched with the regex from the TeX bundle: { name = 'string.other.math.tex'; begin = '$'; end = '$'; swallow = '\$'; ...
On the other hand, the second double dollar signs are matched as the begin regex from the LaTeX bundle: { name = 'string.other.math.block.latex'; begin = '$$'; end = '$$'; },
and the parser goes on to search the entire rest of the document for the closing $$ So what confuses me (and this is what I meant when I asked how it was possible) is how only the first of these is matched by the string.other.math.tex regex, and not the second one as well. How can string.other.math.tex not take priority in both cases? Maybe Allan can answer that, but for right now I see the following possible solutions:
- Make it so that the string.other.math.tex regex needs to match
something between the begin = '$'; end = '$'; Not sure if this is possible. 2) Move the string.other.math.block.latex in the TeX file, and place it above the string.other.math.tex pattern, so that it matches first. 3) Even better, mix the two patterns into one.
Brad
Haris
PS: Out of curiosity, what do you use instead of $$, and how did you historically come to do it that way? In the university of Chicago, I think we all do the displaymath mode that way. In fact, I haven't before met anyone not using it. Are there reasons against using it?
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
On Nov 6, 2005, at 5:23 PM, Brad Miller wrote:
or (the following does not seem to be well documented, but appears to work) $$ dispalyed math equation $$
Hm, I have to admit I find this statement very surprising. It is extremely well documented in the TeXbook: http://www.ctan.org/tex-archive/systems/knuth/tex/texbook.tex I guess this might be the difference: That this is the way to do it in TeX, but not in LaTeX. Anyway...
I don't think there is any difference between equation* and displaymath but I could be wrong...
Well, I guess the question here would be, if you redefine one of the two, does that redefine the other as well? I dunno, I'm afraid I don't have a very deep knowledge of LaTeX either.
I think for purposes of scoping in TextMate we ought to treat them all the same.
I agree.
Upon further investigation of my file, the problem seems to be that all this falls also under the scope of meta.function.with-arg.tex, since it happens within the scope of a \comment{ command. If I close this command before the displaymath part, then it works fine. So this is not a problem simply to do with the math scopes.
Brad
On Nov 6, 2005, at 5:56 PM, Charilaos Skiadas wrote:
On Nov 6, 2005, at 5:23 PM, Brad Miller wrote:
or (the following does not seem to be well documented, but appears to work) $$ dispalyed math equation $$
Hm, I have to admit I find this statement very surprising. It is extremely well documented in the TeXbook: http://www.ctan.org/tex-archive/systems/knuth/tex/texbook.tex I guess this might be the difference: That this is the way to do it in TeX, but not in LaTeX. Anyway...
I don't think there is any difference between equation* and displaymath but I could be wrong...
Well, I guess the question here would be, if you redefine one of the two, does that redefine the other as well? I dunno, I'm afraid I don't have a very deep knowledge of LaTeX either.
I think for purposes of scoping in TextMate we ought to treat them all the same.
I agree.
Upon further investigation of my file, the problem seems to be that all this falls also under the scope of meta.function.with-arg.tex, since it happens within the scope of a \comment{ command. If I close this command before the displaymath part, then it works fine. So this is not a problem simply to do with the math scopes
It Looks like our replies crossed paths on the network...
Next question: \comment appears to be your own block comment function. So, Should any of the latex that follows be scoped as anything other than a comment? Since \comment is your own function the answer is probably no, but how to know what should work and what shouldn't
The problem for me is that it is not clear what LaTeX/TeX constructs are legal to embed in what other constructs. For a simple example, replace \comment with \textbf Strangely, all the text inside the textbf does not bold the text that was previously commented. Nor does it produce an error. Other times I have tried to put latex environments inside TeX commands and they result in errors....
That said, I moved the displaymath declarations into the TeX syntax file and all appears to be scoped correctly. in your document.
I'm going to do a bit more testing and then I'll commit the change.
Brad
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
On Nov 6, 2005, at 6:22 PM, Brad Miller wrote:
Next question: \comment appears to be your own block comment function. So, Should any of the latex that follows be scoped as anything other than a comment? Since \comment is your own function the answer is probably no, but how to know what should work and what shouldn't
The problem for me is that it is not clear what LaTeX/TeX constructs are legal to embed in what other constructs. For a simple example, replace \comment with \textbf Strangely, all the text inside the textbf does not bold the text that was previously commented. Nor does it produce an error. Other times I have tried to put latex environments inside TeX commands and they result in errors....
It's not a question of comments really. The point is that users can define their own commands to do whatever they want, and the arguments to the commands should in theory be allowed to be arbitrary LaTeX code, and hence should be treated as such. For instance I might decide, just as a hypothetical example, to use a command called \double, which just creates two copies of its argument, i.e. \newcommand{\double}[1]{# #1} Then I would expect something like: \double{$x^2+1=0$} to recognize $x^2+1=0$ as math, which it does at the moment. The problem is that these commands are caught by the TeX syntax, so of course can only see stuff from the TeX syntax. Maybe an option would be to have them be caught by the LaTeX syntax also? It would be duplicating code, that's why I don't particularly like it as an approach I want to find the time to look more deeply into the LaTeX bundle stuff, but haven't had the time for it, and won't have for another month. I think for now the best thing is to pile stuff up in the test.tex file. Things work out fine for most people, and most of the times, so we can worry about such details a bit later.
That said, I moved the displaymath declarations into the TeX syntax file and all appears to be scoped correctly. in your document.
Thanks.
I'm going to do a bit more testing and then I'll commit the change.
Brad
Haris
On 07/11/2005, at 1.52, Charilaos Skiadas wrote:
[...] Then I would expect something like: \double{$x^2+1=0$} to recognize $x^2+1=0$ as math, which it does at the moment. The problem is that these commands are caught by the TeX syntax, so of course can only see stuff from the TeX syntax. Maybe an option would be to have them be caught by the LaTeX syntax also? It would be duplicating code, that's why I don't particularly like it as an approach
What can be done is to let the TeX syntax include "$base" for the command argument, that will include the base syntax file, so that'd be LaTeX when the document is marked up as LaTeX, and TeX when the document is TeX.
We use this e.g. in the C grammar, which has a few recursive rules, and is (sometimes) included by the C++ or Objective-C grammar.
Though not sure if there are other problems using the LaTeX grammar recursively (as I got the impression of, from Brads comment).
On Nov 6, 2005, at 10:00 PM, Allan Odgaard wrote:
On 07/11/2005, at 1.52, Charilaos Skiadas wrote:
[...] Then I would expect something like: \double{$x^2+1=0$} to recognize $x^2+1=0$ as math, which it does at the moment. The problem is that these commands are caught by the TeX syntax, so of course can only see stuff from the TeX syntax. Maybe an option would be to have them be caught by the LaTeX syntax also? It would be duplicating code, that's why I don't particularly like it as an approach
What can be done is to let the TeX syntax include "$base" for the command argument, that will include the base syntax file, so that'd be LaTeX when the document is marked up as LaTeX, and TeX when the document is TeX.
Brilliant!!! That's exactly what I was hoping there would be.
We use this e.g. in the C grammar, which has a few recursive rules, and is (sometimes) included by the C++ or Objective-C grammar.
Though not sure if there are other problems using the LaTeX grammar recursively (as I got the impression of, from Brads comment).
Just tried it, and briefly looked at the test.tex file, and it looked ok to me. Maybe Brad can have a closer look at it? I also looked at a bigger latex file I had, and that worked fine too.
I think maybe the problems that Brad was referring to might have had to do with fragile commands and stuff like that. In the sense that lots of LaTeX commands and environments alter things in a way that containing other commands inside the arguments could cause the processor to complain or produce unexpected results. But I don't think they are syntactically wrong, even if they don't compile properly. Or maybe I'm waaay off and Brad's problems had to do with the code not being processed properly.
While we are at it looking at one of my tex-files I encountered another problem, with \citep. The following command: \citep[see][Lemma~II.1.3]{Deligne:1970} catches the 'see' and 'Lemma~II.1.3' together with the \citep. The regexp reads: begin = '(\cite(al)?[tp]?*?([[^]]*]){,2}{)'; end = '(})'; I think I edited that command to catch the brackets, though not knowing how many square bracket arguments there would be (at most two are allowed in this case) I did know how to match them properly. Is there a nice way to do it? I am guessing it should be like the multi- argument function thingie. I am looking at it though, but I can't quite understand how it works. Can you explain a bit the logic for the end pattern? ( end = '(?=\s*[^{[\s])'; )
Haris
On 07/11/2005, at 6.05, Charilaos Skiadas wrote:
[...] the multi-argument function thingie. I am looking at it though, but I can't quite understand how it works. Can you explain a bit the logic for the end pattern? ( end = '(?=\s*[^{[\s])'; )
The begin pattern matches \command and uses look-ahead on [ or {.
Then the sub-rules will match any {…} or […] argument (one rule for each).
The end pattern does a look-ahead on something which is NOT [ or { (or space).
So what happens is, the end pattern will cause the matching to stop as soon as it can match a non-argument starter character (but using look-ahead to not eat it). However, since the sub-rules will match any full argument, the end-pattern won't see what's inside {…} or […].
So for example if we have “\foo{bar}fud” then first \foo is matched by the begin pattern, the rest of the string is now “{bar}fud”. We have 3 candidate rules for this, the two sub patterns and the end- rule. One sub-pattern will match “{bar}”, so we're left with “fud”, where we again have 3 candidates, this time the end-pattern will match it, since it doesn't start with a { or [, and so, we will leave the sub-patterns.
Since I've just given a short course on LaTeX, I had some stuff looked up and on slides (in Dutch). So to be complete, the following are all ways to get math modes in LaTeX:
Brad Miller wrote:
There are four environments that put LaTeX in math mode: math, displaymath, eqnarray, and equation.
These can be used as begin/end keywords, but the following are also possible:
- math: $…$ or (…) - displaymath: $$…$$ or […]
Jeroen.
On Nov 6, 2005, at 5:09 PM, Charilaos Skiadas wrote:
Hi Brad, On Nov 6, 2005, at 4:19 PM, Brad Miller wrote:
On Nov 6, 2005, at 3:54 PM, Charilaos Skiadas wrote:
In other words, the two dollars signs next to each other are matched as the begining and end of a simple math mode, instead of being considered as the beginning of a block math mode. What makes it even weirder is that the following: $$\int_{C_{t}}e^{g(z,t)}f(z,t)\d z$$ where $g,f$
I pasted your example above into the test.tex document and the text between the $$ .. $$ is scoped as string.other.math.tex The word 'where' is scoped as text.latex and the g,f is scoped as string.other.math.tex I believe those are all correct yes?
I think I see the problem. If you check the scope of the word right before the first $$ 'integrals of the form ...' You will see that the scope of those words is meta.function.with-arg.tex This is because all of this text is contained inside the command that starts \comment{My research...
My first question is, is this really what you want?
The meta.function.with-arg.tex rule recursively matches anything inside the { } by including source.tex.
So, by moving the displaymath rule into the TeX syntax file that will help. I have tried recursively including the LaTeX syntax but things got really ugly when I did that.
So, I think it is time for some more work. I need to look at both the TeX and the LaTeX syntax definitions and be very careful about what goes in each. I've always been pretty casual about the difference between the two, but here is an example where that causes problems.
I'll try moving a few things around and see if that helps in the short term anyway.
Brad
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
On Nov 6, 2005, at 6:00 PM, Brad Miller wrote:
I think I see the problem. If you check the scope of the word right before the first $$ 'integrals of the form ...' You will see that the scope of those words is meta.function.with-arg.tex This is because all of this text is contained inside the command that starts \comment{My research...
My first question is, is this really what you want?
Yes, it is a very unusual use of a command I have to admit, but I use it 'comment out' parts that I want there, but not deleted, in case I want them in later on. I just define a command at the beginning: \newcommand{\comment}[1]{} so that it is all omitted out, but if I want it there I'll just change the command to read: \newcommand{\comment}[1]{#1}
The meta.function.with-arg.tex rule recursively matches anything inside the { } by including source.tex.
So, by moving the displaymath rule into the TeX syntax file that will help. I have tried recursively including the LaTeX syntax but things got really ugly when I did that. So, I think it is time for some more work. I need to look at both the TeX and the LaTeX syntax definitions and be very careful about what goes in each. I've always been pretty casual about the difference between the two, but here is an example where that causes problems.
I'll try moving a few things around and see if that helps in the short term anyway.
So, our problem in general is that, inside LaTeX code, we want to allow patterns that appear in the TeX syntax, but in such a way that they understand that they have LaTeX power, i.e. that they can recognize anything in the LaTeX syntax, not just the TeX syntax. Not sure if this is possible as things are right now. I don't think it would be correct to include the LaTeX syntax in TeX either, since if the file is an actual TeX-file and not a LaTeX-file there shouldn't be LaTeX code in it. Not sure what the right way to structure things is. I do like the idea of TeX commands being in their own syntax file, and the other syntaxes inheriting from them, but I am not sure what the best way to implement this would be then. For now I am happy with things being the way they are. This problem doesn't really appear in most regular uses of the bundle.
Brad
Haris
-
Allan Odgaard -
Brad Miller -
Charilaos Skiadas -
Charilaos Skiadas -
Jeroen van der Ham