Hi,
I've been playing with TM 1.5.1 (948) and wanted to see how it handled XML files on my 1.67GHz Powerbook.
The file I needed to mess with was a 3.7MB FO file with some long lines (over 10000 characters and some over 100000.) The formatted objects were generated by an XSL processor which didn't care about long lines ;-)
Because it had a .fo extension it was opened as Plain Text.
Scrolling around would cause a beachball on some of those really long lines. (Soft wrapping was on.)
I used the popup to switch to XML. This caused a beachball for about 2 minutes, and subsequent attempts to scroll often beachballed too. Soft wrapping seems to have got turned off by this switch, which is consistent with the line numbers in the margin.
The big mistake was to try and do a Find/Replace of all ">" with ">\n". Whoah. It's been using all the CPU and beachballing for over 30 minutes now, with no obvious way to interrupt it...
It is a bit distasteful to bring up other products ;-) but TextWrangler did that Find/Replace in about 2 seconds. Going to XML mode in TextWrangler was instantaneous.
Are these known problems? I googled for a bit but couldn't find anyone reporting similar things.
Is there any chance these might get addressed in the 1.5.x codebase? I want to love TM but currently cannot :-(
Cheers,
Chris
On 19/5/06 3:20, Allan Odgaard throw-away-1@macromates.com wrote:
On 19/5/2006, at 7:46, Chris Ridd wrote:
[...] Are these known problems?
I am aware of TextMate¹s performance profile, yes.
I couldn't believe that it wasn't known about, but it was worth checking. BTW the switch of that big file to XML mode was still going after 4 hours :-((
Are these performance problems going to be addressed in the 1.5 code?
[...] I want to love TM but currently cannot :-(
That¹s okay, because TM would not be able to return your love.
Heh. Would it be upset if I saw another editor behind its back?
Cheers,
Chris
On May 19, 2006, at 10:20 AM, Allan Odgaard wrote:
On 19/5/2006, at 7:46, Chris Ridd wrote:
[...] Are these known problems?
I am aware of TextMate’s performance profile, yes.
[...] I want to love TM but currently cannot :-(
That’s okay, because TM would not be able to return your love.
I think this is an interesting response. Here is a product developed heavily upon other open source or freely available software, where the author recommends that users seek support from the user community, and even a very large part of the functionality is written by the user community. Yet TextMate itself is not open source, nor is it free. So when someone asks about performance and whatnot, you can't simply say 'you get what you pay for', because TextMate does cost something. And compared to other similarly complex software, it costs a lot.
I too am concerned about the performance profile. I am also concerned that nearly everything is done with Regular Expressions, which I'm sure are in large part responsible for the performance of the app. Regular Expressions can do an awful lot in the arena of text processing; but a lot of what an editor used by programmers needs to do can not be done at all, or at least very well, by over reliance on Regular Expressions.
So can you share more than a one sentence response on what we can expect in the way of performance, and API going forward? I understand there is a version 2.0 due sometime. Will this new version address the performance profile, and allow for other than RegEx for an API?
Thanks.
Regular Expressions can do an awful lot in the arena of text processing; but a lot of what an editor used by programmers needs to do can not be done at all, or at least very well, by over reliance on Regular Expressions.
While I agree with the desire for ever more powerful mechanisms, I'd like to speak up quickly just to say that what Allan has effectively created with the scope system, used both for styling and for commands/etc. is a real parser -- a CFG or set of CFGs -- not a mere regular expression, because of the ability to nest elements in a language grammar. Just like in a conventional machine language parser, regular expressions are used to recognize individual parse elements, but the structure it generates and uses to analyze and represent the code or other file contents is context-free, and hence just about as powerful as anything used in other editors, even up to the level of Eclipse's fancy-dan refactoring tools and so forth. What could make this more accessible for more advanced tools might be a more direct API to run over (and potentially transform) the parse tree, not just the current context, but what it does now is still much more powerful than mere regular expressions. -jrk
On May 26, 2006, at 5:14 PM, Jonathan Ragan-Kelley wrote:
Regular Expressions can do an awful lot in the arena of text processing; but a lot of what an editor used by programmers needs to do can not be done at all, or at least very well, by over reliance on Regular Expressions.
While I agree with the desire for ever more powerful mechanisms, I'd like to speak up quickly just to say that what Allan has effectively created with the scope system, used both for styling and for commands/etc. is a real parser -- a CFG or set of CFGs -- not a mere regular expression, because of the ability to nest elements in a language grammar. Just like in a conventional machine language parser, regular expressions are used to recognize individual parse elements, but the structure it generates and uses to analyze and represent the code or other file contents is context-free, and hence just about as powerful as anything used in other editors, even up to the level of Eclipse's fancy-dan refactoring tools and so forth. What could make this more accessible for more advanced tools might be a more direct API to run over (and potentially transform) the parse tree, not just the current context, but what it does now is still much more powerful than mere regular expressions. -jrk
While in general I agree with this, Ill make two points:
1) saying it's up to par with Eclipse is just laughable, and so self- evidently false that I'll leave it at that. Anyone wanting to debate the finer points of this are welcome to, but I'm not sure the list is up to it. :) Just e-mail me directly.
2) I have no problem calling it more that RegEx; call it what you will. It is powerful, and can do wonders. I'm impressed with it's abilities. But. It is still BUILT upon RegEx, and runs a RegEx engine, and therefore, is going to be much slower that what Eclipse, Visual SlickEdit, and BBEdit provide (just the editors of these products; not the full products).
- saying it's up to par with Eclipse is just laughable, and so
self-evidently false that I'll leave it at that. Anyone wanting to debate the finer points of this are welcome to, but I'm not sure the list is up to it. :) Just e-mail me directly.
- I have no problem calling it more that RegEx; call it what you
will. It is powerful, and can do wonders. I'm impressed with it's abilities. But. It is still BUILT upon RegEx, and runs a RegEx engine, and therefore, is going to be much slower that what Eclipse, Visual SlickEdit, and BBEdit provide (just the editors of these products; not the full products).
So, comparing TextMate with Eclipse? Actually I started a job just some weeks ago and have to work with Eclipse/Win. (Unfortunately not Java, but PHP mostly). So there is no refactoring of any kind, funny code completion (it scans the whole project and lets you select an implementation of a method that is defined anywhere when it could really just open the implementation of the parent class I'm calling). Eclipse has no means of macros whatsoever, lacks all the advanced editing, customization and enhancement (ok, easy enhancement, tried to write a serious eclipse plugin once?) options of TextMate?
And as you laugh about the comparison on point one, I can only shake my head about your statement two. Just go and write an incremental compiler for all the languages TextMate supports and come back to us here. Also compare the age and maturity of TextMate to, say BBEdit. And how easily the user can enhance it. Sure it's not open source (you pointed that out earlier) but it does not have IBM and several other major players behind it, but one man. Your criticism in honour, that is always welcome, but you should be a little considerate of these things.
All in all, TextMate is surely not perfect, nor is any other editor. The performance issues bit me at some points, like when trying to open a colloquy log (iirc) which saves it as xml, all on one line, so I had a million characters sitting there. And that is an issue of how the text/glyphs/whatever is rendered by TextMate, because of ATSUI. I'm not into that, but I can understand when the developer says, he can't simply change the way of how everything is displayed. And this has come up in the past, so you should have searched the list. Back then we were also told about future directions and the internal editing component will be the revamped in the future. All of that is findable on this list, I'm sure. The version numbers which were talked about back then, are no longer acurate though.
Good night, Soryu.
On 27/5/2006, at 1:54, Timothy Reaves wrote:
[...] It is still BUILT upon RegEx, and runs a RegEx engine, and therefore, is going to be much slower
FYI roughly half the time spent by the parser is spent in CoreFoundation because I currently use NSArray and NSDictionary to represent the grammar. So regular expressions are not the culprits you try to make them out to be.
But when speaking of the parser and other text editors (that you bring up): the performance difference is a) psychological because TextMate does not redraw the first page of the document before the entire document has been parsed b) because TextMate does something very different compared to the editors you mention and c) it generally has more complex grammars (try put PHP in an HTML here doc in a ruby script embedded in Erb tags in an HTML file -- unlike other text editors, TextMate handles this gracefully.)
As Jonathan said, this is a parser. It parses your document into a tree and maintains that tree when you edit the text. This tree preserves information obtained during parsing, and styling is done using a style sheet with a selector based rule system which works on this tree.
So your comparison with other editors makes little sense. TextMate has a powerful declarative grammar system which allows the user to describe how his code should be parsed, assign functionality and apply preferences to user-defined semantic units based on this, coincidentally this system is also used for styling. It is the performance of this system you seem to be criticizing, stating that other editors are much faster -- but other editors do not do this, they don’t do anything like it! Even ignoring this sophisticated architecture for rule based behavior adjustments and focusing alone on the syntax highlight: other editors are still not as sophisticated, case in point: John Gruber (creator of Markdown) worked for one of the companies which product you mentioned, still that product is not capable of providing syntax highlight for his Markdown notation.
As for switching to a more API-based system: No, I am not doing that (at least not anytime soon) because this has a major flexibility and maintenance drawback.
The current system has been incrementally improved with new features, this has been possible exactly because the system is declarative. For language grammars, there is just one parser which use the declarations, instead of a dozen plug-ins each with their own code -- having separate plug-ins makes mixing grammars rather problematic (how TM supports embedded languages), and take something like the planned grammar injection based on scope selectors: that would certainly not have been possible in an API-based system without having to rewrite all existing plug-ins -- and even coming up with a proper API for this is quite a challenge.
So while I will gladly share my plans of improving this system in the future, it requires that you actually understand the system.
All that said, the problems described by the OP are primarily caused by other things than the language parser. I do not view the language parser as something which badly affects TextMate’s performance -- there are half a dozen other things which do affect the performance of TextMate in a bad way (i.e. scales badly, the language parser scales just fine [1]), and I will gradually improve these things. The language parser however will see improvements mainly in functionality.
[1] An exception being that editing a line requires the line to have been parsed before the key can be resolved (due to dependency on scope for key equivalents), and editing a line causes the entire line to be re-parsed, making editing a line linear in time with its length (where the factor here is parsing a line) -- this is generally not a problem, because I also redraw the entire line, and that is a much larger factor, but an exception is when enabling soft wrap, where then only the logical line is redrawn, but the physical line is re- parsed.
On 26/5/2006, at 22:46, Timothy Reaves wrote:
[...] So when someone asks about performance and whatnot, you can't simply say 'you get what you pay for'
While I personally would not use such phrase, I don’t see why you can’t say such thing about TextMate.
You pay an indie software developer €39 for a shareware application. Did you not get a shareware application worth €39? Let me remind you that hiring a software developer to do custom development for you will likely cost €100 or more per hour (and I doubt you can find anyone who can write something like TextMate in less than a year), and it’s not uncommon to pay for support and/or upgrades. I currently provide that as a free bonus, and even do semi-custom development for you guys.
I do this of own choosing, so it is not because I expect any special gratitude for this gesture, but frankly your statement does seem pretty out of place!
[...] compared to other similarly complex software, it costs a lot.
No it does not. Commercial/shareware software of the same complexity costs the same or more, sometimes much more. F/OSS is hard to compare, but if you think of the high-profile projects like Eclipse or Firefox, those projects are backed by companies who invest millions of dollars in them (lookup the price for being a Mozilla Foundation), even governments have put money into the Mozilla project -- and those money do come from somewhere.
Allen,
Amen :) Regardless, I love TextMate and the Job you've done, and I'd gladly pay for it again.
Eric Coleman
On May 26, 2006, at 10:40 PM, Allan Odgaard wrote:
On 26/5/2006, at 22:46, Timothy Reaves wrote:
[...] So when someone asks about performance and whatnot, you can't simply say 'you get what you pay for'
While I personally would not use such phrase, I don’t see why you can’t say such thing about TextMate.
You pay an indie software developer €39 for a shareware application. Did you not get a shareware application worth €39? Let me remind you that hiring a software developer to do custom development for you will likely cost €100 or more per hour (and I doubt you can find anyone who can write something like TextMate in less than a year), and it’s not uncommon to pay for support and/or upgrades. I currently provide that as a free bonus, and even do semi-custom development for you guys.
I do this of own choosing, so it is not because I expect any special gratitude for this gesture, but frankly your statement does seem pretty out of place!
[...] compared to other similarly complex software, it costs a lot.
No it does not. Commercial/shareware software of the same complexity costs the same or more, sometimes much more. F/OSS is hard to compare, but if you think of the high-profile projects like Eclipse or Firefox, those projects are backed by companies who invest millions of dollars in them (lookup the price for being a Mozilla Foundation), even governments have put money into the Mozilla project -- and those money do come from somewhere.
For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate
On 27 May 2006, at 03:40, Allan Odgaard wrote:
I do this of own choosing, so it is not because I expect any special gratitude for this gesture, but frankly your statement does seem pretty out of place!
I agree. I can't see how any rational person could look at the quality of TextMate and the amount of work that's clearly gone into it and consider it over priced. The parser at its core makes it unique to the extent that the only real problem I have with TextMate is that it makes it almost unbearable to use other editors now.
Of course this mailing list gets a bit tedious if we just spend all our time telling you what a great product TM is - but if you're working alone you need all the support and feedback you can get.
Thanks for a great product Allan.
On 26 maj 2006, at 22.46, Timothy Reaves wrote:
On May 19, 2006, at 10:20 AM, Allan Odgaard wrote:
On 19/5/2006, at 7:46, Chris Ridd wrote:
[...] Are these known problems?
I am aware of TextMate’s performance profile, yes.
[...] I want to love TM but currently cannot :-(
That’s okay, because TM would not be able to return your love.
I think this is an interesting response. Here is a product developed heavily upon other open source or freely available software, where the author recommends that users seek support from the user community, and even a very large part of the functionality is written by the user community. Yet TextMate itself is not open source, nor is it free. So when someone asks about performance and whatnot, you can't simply say 'you get what you pay for', because TextMate does cost something. And compared to other similarly complex software, it costs a lot.
Really, you think this costs a lot, i would say that TextMate is cheap as hell. And alot of what drew me to it was the fact that there was such an active community around it developing for it. Now that i think of it that was the same reason that i started using emacs a long time ago. And ofc Alan can say 'you get what you pay for', although i dont think that is what he said, since you have a trial period before buying TextMate to decide whetever to buy it or not. And regarding the performance, a search on the mailing list would had given him the answer to it.
I too am concerned about the performance profile. I am also concerned that nearly everything is done with Regular Expressions, which I'm sure are in large part responsible for the performance of the app. Regular Expressions can do an awful lot in the arena of text processing; but a lot of what an editor used by programmers needs to do can not be done at all, or at least very well, by over reliance on Regular Expressions.
Like what, give some examples please.
So can you share more than a one sentence response on what we can expect in the way of performance, and API going forward? I understand there is a version 2.0 due sometime. Will this new version address the performance profile, and allow for other than RegEx for an API?
What kind of API do you want and what would it do that the current cant do?
/Mikael
-
Allan Odgaard -
Andy Armstrong -
Chris Ridd -
Eric Coleman -
Jonathan Ragan-Kelley -
Mikael Hultgren -
Soryu -
Timothy Reaves