On Sep 11, 2007, at 1:29 AM, Alain Matthes wrote:

Le 11 sept. 07 à 03:11, Brad Miller a écrit :

...

Then I don't understand very well why all the part about "lstlisting"

"lstlisting" don't work with utf8 and why it is not possible to make this work outside text.tex.latex ? (perhaps it is not possible ?) I suppose I am partially to blame for that, since I worked on the grammar early on, and I do a lot with lstlisting in my writing. Why is this a problem for you?

hello

No you are not to blame and I want to say that your work is fine and it's a real pleasure to work with TexTMate on LaTeX. But I now I've a little time to see what there is in the grammar and to make a theme for LaTeX.

I know that "lstlisting" is sometimes very important for authors but we can say the same for a lot of packages. My english is poor so perhaps my question is difficult to understand :

I would to know how is managed the bundle :

There parts like

latex tex tex.math

now if I want to create an extension for that with

grammar (keywords) for pgf/tikz or calc,ifthen,xkeyval ( for those who make packages)% i'm not sure of my english here !! pstricks etc...

I would like to know if you have a method (project ??) and how to add grammar .

I can make my personnal latex bundle but i would prefer to make something useful for every one (it's possible!)

Regards Alain

Though I'm still not entirely sure what the question is, I'll try to answer it.

The LaTeX bundle contains the following grammars:

TeX TeX Math LaTeX LaTeX Beamer LaTex Memoir

1) The TeX grammar contains items that should be valid in any system based on TeX. (For instance a grammar for ConTeXt should probably be including this grammar.) 2) The TeX Math grammar contains commands that are used in math mode. It is included in other grammars whenever they have to enter math mode. Typically one would create just a repository rule in a grammar for something like that, instead of a brand new language, but we needed to be able to call this grammar from lots of other grammars, so we had to make it into a separate grammar. 3) The LaTeX grammar is the basic grammar. It includes the TeX grammar and adds all the LaTeX specific stuff. In particular, it adds the [ ... ] and ( ... ) math modes, and hence needs direct access to the TeX Math bundle, which it includes at those points. (This is why we could not simply have TeX Math as a repository in the TeX grammar, because then the LaTeX grammar couldn't call it.) 4) For specialized document classes, at this point in time, we simply create a new grammar that includes the LaTeX grammar and adds its own specific stuff. (This might become simpler/better in TM 2.0). Examples of this are the Beamer and Memoir grammars.

5) For commands in particular packages, we typically would add those to the LaTeX grammar. Just make the additions you want to the grammar and send it to us, and we would add them. (Or consult with us on what changes you think are needed.). So for adding specific package commands, this is the option you should aim for.

6) For working on sty files however, ideally we would have a different grammar. This grammar would include the LaTeX grammar in it, and add all the stuff that package authors would need. So for a "package authors" grammar, this is the option you should aim for.

Hope this helps you get started, please feel free to ask more questions. The structure of the LaTeX bundle is admittedly not the simplest in the world, and the vastness and free structure of LaTeX is probably partly to blame.

Haris Skiadas Department of Mathematics and Computer Science Hanover College

Alain,

Thanks for the good ideas!

So I think what you are saying is that it would be nice to conditionally include some grammars as a part of the latex grammar depending on which packages you are using. This would have some very good benefits: 1. It would keep package grammars like lstlistings from making the plain latex grammar overly long. 2. This might have some speed benefits for editing as well. 3. It would also make it easy for people to incrementally add new scopes to the grammar that are specific to individual packages without having to modify the plain latex grammar.

This is great. But as far as I know this is not possible in the current version of TM.

Today we have to do things the other way around For example I would have to make an lstlistings grammar that included latex and tex. But this is not good at all since it only would allow me to have the specialized grammar for one particular package.

So, until we get to TM 2.0 or later, or someone tells us how to conditionally include grammars, we have to keep adding to the latex grammar for each package that we want to have scoped.

Brad

On 9/11/07, Alain Matthes alain.matthes@mac.com wrote:

But why conditionally include grammars ??

Because then you could base which grammars are included based on which packages are included in each document. Suppose that this actually worked:

{ name = 'meta.package.latex'; match = '^\usepackage{lstlisting}' patterns = ( { include = 'text.tex.latex.lstlisting'; } ); },

or even better { name = 'meta.package.latex'; match = '^\usepackage{(lstlisting)}' patterns = ( { include = 'text.tex.latex.$1'; } ); },

I'm not a specialist of script, regex etc.... but in a first time

it's possible to build some grammars like

text.tex text.tex.math text.tex.beamer text.tex.listings text.tex.tikz etc....

And to finish the text.tex.latex by some include like

include = 'text.tex.listings'; ?

Each user can modify the list of the package grammars !

I think this is where we differ. I don't think we want each user to be modifying the latex grammar to decide which packages to include. For one thing it will be different for each different project. So, I"m either going to end up including everything all the time, or I'm going to drive myself crazy modifying the latex grammar every time I work on a different writing project. Plus most users don't want to go anywhere near editing the grammar file!

About listings, why make a grammar to this package ?

Because lstlistings is the best package I've found for including source code, and since I'm writing CS textbooks, I do that a lot. If you can recommend a better one I would be happy to look at it! But really, most of the grammar for the package is only concerned with then command and then including the grammar for the particular source code that is inside the lstlistings environment.

There is a way to turn around the difficulties.

\lstinputlisting[key=value list]{hfile name}

example :

\usepackage{listings} \lstset{language=C} ... \begin{document} \lstinputlisting{main.c} \end{document}

or you can make that

\lstinputlisting[first=10,last=50]{main.c}

I know about this feature of lstlisting but I like to have the code right there to look at as I'm writing. I guess its a writing style issue but I tried your method for a while and I did not like it at all.

It is the only way for me to use listings because I'm french (I need

only to include latex code but with accents) and it's impossible to use listings with utf8 actually, so I need to save file with latin1 encoding and to use two encodings latin1 and utf8 is the same file ! (very bad)

I don't understand about the problem with utf8. My latex source files are all encoded as utf8 documents and I don't have any problem with lstlistings.

I suppose that you need to include java code or python code but there

are scopes to verify the syntax of these languages. So why waste time with a difficult grammar and not include the sources of codes.

Again, the lstlisting grammar includes source.java or source.python. In fact by my calculations the number of lines in the latex grammar dedicated to the listing package is only 47 lines out of 494 total.

Regards Alain

---

For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate

On Sep 11, 2007, at 1:00 PM, Alain Matthes wrote:

Le 11 sept. 07 à 13:23, Charilaos Skiadas a écrit :

...

3. The LaTeX grammar is the basic grammar. It includes the TeX

grammar and adds all the LaTeX specific stuff. In particular, it adds the [ ... ] and ( ... ) math modes, and hence needs direct access to the TeX Math bundle, which it includes at those points. (This is why we could not simply have TeX Math as a repository in the TeX grammar, because then the LaTeX grammar couldn't call it.)

Firstly, Thanks for this answer. You understand exactly my problem

I need some precisions about patterns and include:

include = 'text.tex'; i understand this one but the next are not very clear for me

include = '$self';

include = '$base';

$self and $base are very subtle. The idea is the following. Let's consider a rule in the TeX grammar, for instance:

{ name = 'meta.group.braces.tex'; begin = '{'; end = '}'; captures = { 0 = { name = 'punctuation.section.group.tex'; }; }; patterns = ( { include = '$base'; } ); },

Now this rule is supposed to match pairs of braces. Now, inside the braces everything TeX would go. So with pattern = ( {include= '$base'; } ); we tell TM that between the begin and end part it should use all the rules in the TeX bundle for it.

Strictly speaking, I just lied. This is what $self would do. To understand the difference, let's move one step further. We are now in the LaTeX grammar, and we include the TeX grammar in it. Now the TM parser encounters the open brace. It looks it the LaTeX grammar rules, finds nothing, and then moves on to the TeX grammar, where it finds the above rule. It start matching the rule, and now needs to know what is allowed to be used between the braces:

1) $base means everything from the original, LaTeX, bundle (which includes the TeX bundle). 2) $self means only stuff from the TeX bundle (where the rule occured).

In most cases in fact $base is what we want, I think most occurences of $self in the LaTeX grammars should be changed to $base.

...

4. For specialized document classes, at this point in time, we

simply create a new grammar that includes the LaTeX grammar and adds its own specific stuff. (This might become simpler/better in TM 2.0). Examples of this are the Beamer and Memoir grammars.

Perhaps I need to wait for TM 2 !

Questions about Beamer

1. In text.tex.latex we have : firstLineMatch = '^\documentclass

(?!.*{beamer})';

and in 'text.tex.latex.beamer : firstLineMatch = '^\documentclass(\ [.*])?{beamer}';

Right, Beamer files are distinguished by the occurence of "\documentclass{beamer}" in the first line (ideally it should be the first 20 lines or something like that, but we can't do this atm). So we explicitly rule out beamer as a class name from the plain text.tex.latex, so that it gets picked up by the Beamer grammar. We should probably add memoir there.

...

5. For commands in particular packages, we typically would add

those to the LaTeX grammar.

I'm not sure that's a good way because i don't use listings (pb with utf8) and I prefer pgf/tikz to pstricks. I would like to know if it's possible to enable or disable some parts. I'm not sure but I think that's a big and long grammar is not good for the speed to match a long text. That is why , I made a remark about "listings" because it's not "pure" latex.

There is now an excellent way for the preferences with a panel, we can imagine a panel for the grammar package to add : beamer, memoir, xcolor, listings, pstricks, tikz, amsmath etc... and for the author : calc ifthen etex multido

Or a file in the preferences

There is at the moment no easy way to dynamically include grammars. The speed hit from a large grammar is actually minimal, I believe. Also we should separate between classes and packages. Packages are just like libraries in other languages, the commands they provide should just be part of the language (The equivalent in Ruby would be requiring a special language just so that we can match the methods of the String class, or something like that). Also, some packages/ classes automatically load other packages, and it would be hard to detect that and "switch mode".

Do you really want, whenever you have a document and you add a new \usepackage to it, to have to go to a preferences setting and tell TM that for this file you want it to recognize the keywords from this particular package? TM should be just recognizing the keywords right away.

...

Just make the additions you want to the grammar and send it to us, and we would add them. (Or consult with us on what changes you think are needed.). So for adding specific package commands, this is the option you should aim for.

for text.tex.latex

I've make somme additions :

{name = 'meta.preamble.latex'; contentName = 'support.class.latex'; begin = '((\)(?:usepackage|documentclass|RequirePackage| usetikzlibrary))(?:([)([^]]*)(]))?({)';

and some modifications

{name = 'constant.numeric.math.latex';
match = '((\+|-)?)(([0-9]*[\.][0-9]+)|[0-9]+)';

because There is constant.numeric.math.tex and no onstant.numeric.math.latex and i want the sign + or - in the same color !

for text.tex

With keyword.control.tex, I've a problem with some terms like \if@tkz@visible or \ifTKZ@tkzInit@NO . The symbol @ is a problem and in a package author there a lot of terms with @.

I add name = 'keyword.operator.tex'

with begingroup, endgroup, global def edef xdef gdef expandafter newbox newdimen newcount advance multiply divide etc...

but i'm not sure like keyword.control.tex ( with if else fi) that's necessary to a classic user of TeX.

Thanks for these additions, I'll get them in when I get a chance.

When you include text.tex in text.tex.latex, it's not a good thing. there are good package to avoid the use of if etc... for example ifthen

I am not following you here, why is it not a good thing?

A latex user must use \ifthenelse \newboolean \setboolean \isodd \whiledo \equal \lengthtest all these macros are in Latex and not in tex.

Oh these are not tex? Ok, we can fix those, is that the complete list?

Important keywords in LaTex are newcommand, renewcommand, newcounter, setcounter, newlength, setlength, addtolength and some keywords tex are also necessary : advance multiply divide etc...

Basically I don't have a complete list of the commands in LaTeX, so we just need to keep on adding on our existing list.

I'm sorry because a lot of ideas are not in a good order.

Regards Alain

Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Sep 11, 2007, at 4:17 PM, Alain Matthes wrote:

Le 11 sept. 07 à 20:52, Charilaos Skiadas a écrit :

...

...

When you include text.tex in text.tex.latex, it's not a good thing. there are good package to avoid the use of if etc... for example ifthen

I am not following you here, why is it not a good thing?

Sorry but here it'not very easy with my poor english language. In fr.comp.text.tex we have a lot of debates about this.

When you use Latex, it is preferable to avoid to use certain macros of Tex (User' guide of Leslie Lamport page 204).

There is a difference between "something being preferable" and "something being invalid". If using TeX macros in LaTeX is *valid*, then our grammar should (try to) support them, regardless of whether it is recommended to do so or not. We are not trying to force users to use good code, we are merely trying to color their code as best as possible. That's why we are including the TeX grammar.

The first answer is the syntax, the second, some macros TeX are different with the same name in tex and latex.

For example tex : newcount latex : \newcounter{} tex : newdimen latex \newlength{}

And latex provides some other macros \setcounter{}{} \setlength{}{} \addtocounter{}{} \addtolength{}{} etc...

With ifthen package you have \ifthenelse{test}{true code}{false code}

you have \equal{}{} \lengthtest{}{} \boolean{} \newboolean{} and \setboolean{}{} \isodd \whiledo{}{} \and \or \not ( )

I think it's important to have this package in the language grammar.

Ok, we need to include it then, it shouldn't be hard to do. Where would I find its documentation?

...

...

A latex user must use \ifthenelse \newboolean \setboolean \isodd \whiledo \equal \lengthtest all these macros are in Latex and not in tex.

Oh these are not tex? Ok, we can fix those, is that the complete list?

I have begin a list of all the macros of TeX and LaTeX with the most important packages, if you want I can send it to you

Please do.

...

...

Important keywords in LaTex are newcommand, renewcommand, newcounter, setcounter, newlength, setlength, addtolength and some keywords tex are also necessary : advance multiply divide etc...

Basically I don't have a complete list of the commands in LaTeX, so we just need to keep on adding on our existing list.

What is the best way to send the lists of macros (useful also for the completion)

Just send them as an attached file, any format you choose will be fine, ideally make it clear what package each command is in, and whether a package automatically loads another package etc. We'll have to figure out some nice completion way since there are too many names probably, but we'll see.

Regards Alain

Regarding a different grammar for each package: Most packages will really add a very small number of lines, i.e. they would have a list of commands and that's it. I'm not even sure we need a different rule for each package really in most cases.

All this only concerns the syntax, i.e. what words will be colored and how. It has nothing to do with completion and such. This would be a different mechanism, and it would indeed be nice for this mechanism to recognize which packages you include in the text, and only offer completion for commands that would be in those packages. Should not be too hard to do I don't think, if we have a list of commands and what packages they come from.

Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Sep 11, 2007, at 6:40 PM, Alain Matthes wrote:

Le 11 sept. 07 à 23:57, Charilaos Skiadas a écrit :

...

There is a difference between "something being preferable" and "something being invalid". If using TeX macros in LaTeX is *valid*, then our grammar should (try to) support them, regardless of whether it is recommended to do so or not. We are not trying to force users to use good code, we are merely trying to color their code as best as possible. That's why we are including the TeX grammar.

Yes I understand this point of view

...

...

The first answer is the syntax, the second, some macros TeX are different with the same name in tex and latex.

For example tex : newcount latex : \newcounter{} tex : newdimen latex \newlength{}

Ok, we need to include it then, it shouldn't be hard to do. Where would I find its documentation?

Latex Companion version II

Well, I have that, does it offer an exhaustive list of all the commands in the package, and their syntax? Where would I look?

The other question also is, what would you want the grammar to do, more than what happens now? In your example below, the functions are matched and colored (albeit they all get the same color). What else would you like to see?

...

...

Just send them as an attached file, any format you choose will be fine, ideally make it clear what package each command is in,

Ok i will make this for the next week

...

Regarding a different grammar for each package: Most packages will really add a very small number of lines, i.e. they would have a list of commands and that's it. I'm not even sure we need a different rule for each package really in most cases.

Yes you are right but only two packages are very long (pstricks, pgf/tikz) with a syntax very different and sometimes complicated.

Perhaps we could have those two as separate grammars, and include them via the include mechanism.

look at this code of my next package and look at it with the actual syntax. It's not very fine...

ouch!

In any case, I really think we need a dedicated grammar for package files, that would include the LaTeX grammar.

At some point I need to sit down and have a look at all this wonderful work you've been doing with pgf/tikz.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \define@cmdkey [TKZ] {Compass} {style}{} \define@cmdkey [TKZ] {Compass} {lw}{} \define@cmdkey [TKZ] {Compass} {color}{} \define@cmdkey [TKZ] {Compass} {delta}{} \define@boolkey[TKZ] {Compass} {clock}[true]{} \presetkeys [TKZ] {Compass} {% style = solid,% lw = .4pt,% color = black,% delta = 10,% clock = false}{}

\newcommand*{\tkzCompass}[1][]{\tkz@Compass[#1]}% \def\tkz@Compass[#1](#2,#3){% \setkeys[TKZ]{Compass}{#1}% \pgfpointdiff{\pgfpointanchor{#2}{center}} {\pgfpointanchor{#3}{center}} \tkz@ax=\pgf@x% \tkz@ay=\pgf@y% \pgfmathveclen{\pgf@x}{\pgf@y} \edef\tkztempLen{\pgfmathresult}% \pgfpointnormalised{\tkz@ax,\tkz@ay} \pgfmathparse{atan(\pgf@y/\pgf@x)} \edef\tkz@GammaD{\pgfmathresult}% \ifTKZ@Compass@clock% \pgfmathparse{\tkz@GammaD+180}\edef\tkz@GammaD{\pgfmathresult} \fi \draw[shift ={(#2)},% \cmdTKZ@Compass@style,% \cmdTKZ@Compass@color,% line width = \cmdTKZ@Compass@lw]% (\tkz@GammaD-\cmdTKZ@Compass@delta:\tkztempLen pt)% arc (\tkz@GammaD-\cmdTKZ@Compass@delta:% \tkz@GammaD+\cmdTKZ@Compass@delta:% \tkztempLen pt); } %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Regards Alain

Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Sep 12, 2007, at 1:20 AM, Alain Matthes wrote:

In a first time, we can forget the packages like pstricks and pgf. The problem actually is all the functions have the same colour.

The problem of the color is a slightly different one. They are the same color because, syntactically, they are the same thing. Even if we distinguish them among their different packages, we will also need to customize the various themes to make the functions have different color depending on what package they are from.

You see, the problem is that LaTeX goes in some sense against TM's syntax coloring philosophy. TM tends to color things depending on their syntactic meaning. In LaTeX most everything is a function, so they are of course all colored as functions ;).

So we will need to figure out what principle we want to follow when coloring functions different from each other. I.e. how will we group the functions together? Should all functions in a package have the same color? Is there another division more meaningful?

Haris Skiadas Department of Mathematics and Computer Science Hanover College