Remember how Ruby 1.9.3 caused a segfault in RubyMate's catch_exception.rb? Now Ruby 2.0.0 is causing further trouble, on this line:
io = IO.for_fd(ENV['TM_ERROR_FD'].to_i)
This generates "Bad file descriptor". So this means that once again there's bad output whenever an exception is to be displayed.
I realize that TextMate 1 is not quite the thing these days, but I'm still using it and perhaps someone has some thoughts on how I might fix this? I don't know how TM_ERROR_FD is set or what might be wrong with it as a file descriptor (it's just a number).
Thx - m.
On Mar 6, 2013, at 20:21, Matt Neuburg matt@tidbits.com wrote:
[…] Ruby 2.0.0 is causing further trouble, on this line:
io = IO.for_fd(ENV['TM_ERROR_FD'].to_i)
This generates "Bad file descriptor".
What about this:
IO.for_fd(2) << "test\n"
Does that work?
If it works, try this code as well:
rd, wr = IO.pipe fork do io = IO.for_fd(wr.to_i) io << "test\n" io.close end puts rd.gets
[…] I don't know how TM_ERROR_FD is set or what might be wrong with it as a file descriptor (it's just a number).
It is setup by TextMate.executor and the number is the file descriptor which is the write-end of a pipe created.
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 6, 2013, at 20:21, Matt Neuburg matt@tidbits.com wrote:
Ruby 2.0.0 is causing further trouble, on this line:
io = IO.for_fd(ENV['TM_ERROR_FD'].to_i)
This generates "Bad file descriptor".
What about this:
IO.for_fd(2) << "test\n"Does that work?
If it works, try this code as well:
rd, wr = IO.pipe fork do io = IO.for_fd(wr.to_i) io << "test\n" io.close end puts rd.getsdon't know how TM_ERROR_FD is set or what might be wrong with it as a file descriptor (it's just a number).
It is setup by TextMate.executor and the number is the file descriptor which is the write-end of a pipe created.
Very cool, thanks! The problem is that that doesn't fit in with RubyMate's way of processing the error message. We are in SharedSupport/Bundles/Ruby.tmbundle/Support/RubyMate/catch_exception.rb. Here, the error message is parsed and is wrapped in some HTML before being fed into the IO object whose number comes from TM_ERROR_FD. That IO object is evidently piped into TextMate's output window, where it appears formatted. If we feed the HTML to just any old IO (such as 2), it isn't formatted; it appears as raw HTML in the output window.
It's odd, isn't it? What could there possibly be about Ruby 2.0.0 that would make the int value of TM_ERROR_FD a "bad file descriptor" when it is a perfectly good file descriptor under previous versions of Ruby?
I tried just running this script as a way of seeing what file descriptors are good and what file descriptors are bad:
(-200..200).each do |n| begin IO.for_fd(n) << "#{n} test\n" puts "#{n} succeeded" rescue puts "#{n} failed" end end
Only 1 and 2 are good. But TM_ERROR_FD is 9 (though that particular number is just an implementation detail on any given run).
Mysterious, eh?
By the way, I see now that this same question has been asked before:
http://lists.macromates.com/textmate/2012-April/034878.html
m.
On Mar 7, 2013, at 5:45 PM, Matt Neuburg matt@tidbits.com wrote:
Allan Odgaard wrote:
If it works, try this code as well:
rd, wr = IO.pipe fork do io = IO.for_fd(wr.to_i) io << "test\n" io.close end puts rd.gets
Did you run the above with Ruby 2.0?
[…] If we feed the HTML to just any old IO (such as 2), it isn't formatted; it appears as raw HTML in the output window.
Yes, “RubyMate” will treat data from your program’s stdout (and stderr) as “raw” and in need of escaping. However, it creates a pipe that can be written to, which should be HTML. The TM_ERROR_FD is the “file descriptor” pointing to the write-end of this pipe.
[…] I tried just running this script as a way of seeing what file descriptors are good and what file descriptors are bad: […]
Basically file descriptors are represented via integers, but the integer is just an index into a table of open files/pipes. So whether or not a given integer is a valid file descriptor depends on wether or not there is an entry in the process’ table.
File descriptor 0, 1, and 2 though are normally setup as stdin, stdout, and stderr.
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 7, 2013, at 5:45 PM, Matt Neuburg matt@tidbits.com wrote:
Allan Odgaard wrote:
If it works, try this code as well:
rd, wr = IO.pipe fork do io = IO.for_fd(wr.to_i) io << "test\n" io.close end puts rd.gets
Did you run the above with Ruby 2.0?
Sure, and it outputs "test", but I really don't quite see what it has to with what I asked.
The problem I'm trying to solve is that any time there's an exception when running a Ruby script under TextMate, we wind up in RubyMate's catch_exception.rb, which starts like this:
at_exit do if (e = $!) && !e.instance_of?(SystemExit) require "#{ENV['TM_SUPPORT_PATH']}/lib/escape"
io = IO.for_fd(ENV['TM_ERROR_FD'].to_i)
io.write "<div id='exception_report' class='framed'>\n"
Etcetera. In Ruby 2.0 (but not in Ruby 1.8.7 or Ruby 1.9.3) we are failing on the "for_fd" line. The question is what to do about that.
If I substitute 2, as I said, I get output from the exception, sure, but it's displayed as unformatted raw HTML, which is not the same as what happens under 1.8.7 or 1.9.3. Moreover, under all versions of Ruby, TM_ERROR_FD is some high number such as 11, so the question remains why this is a valid parameter to "for_fd" under Ruby 1.8.7 and 1.9.3 but not Ruby 2.0.
Basically file descriptors are represented via integers, but the integer is just an index into a table of open files/pipes. So whether or not a given integer is a valid file descriptor depends on wether or not there is an entry in the process' table.
Yes, and the issue is why TM_ERROR_FD (which might be, say 11) is valid during catch_exception.rb's at_exit in Ruby 1.8.7 and 1.9.3 but not in Ruby 2.0. I was hoping you might have some idea about this. Could it be that the TM_ERROR_FD pipe created in executor.rb is somehow not surviving? Is it being closed too soon somehow? But why would that be the case in Ruby 2.0 only? Maybe that's the mystery that needs solving here... I'm not knowledgeable enough to experiment with executor.rb. m.
On Mar 8, 2013, at 3:59, Matt Neuburg wrote:
Allan Odgaard wrote:
[…] Did you run the above with Ruby 2.0?
Sure, and it outputs "test", but I really don't quite see what it has to with what I asked.
It is testing if the core functionality that TextMate.executor relies on is available and works in ruby 2.0.
[…] In Ruby 2.0 (but not in Ruby 1.8.7 or Ruby 1.9.3) we are failing on the "for_fd" line. The question is what to do about that.
Figure out which assumption 2.0 breaks, which is what I was trying to do by having you run the code that tests our core assumptions (creating a pipe and getting the write-end by creating a file descriptor from its integer representation).
[…] Could it be that the TM_ERROR_FD pipe created in executor.rb is somehow not surviving? Is it being closed too soon somehow?
It does look like it is being closed, though _nothing_ in the user script should close this file descriptor, so the most likely explanation would be, that ruby 2.0 closes all open file descriptors (except 0-2) and _before_ handling at_exit code.
I find this a little weird though, if they are actually doing that — should be possible to test though.
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 8, 2013, at 3:59, Matt Neuburg wrote:
Allan Odgaard wrote:
Did you run the above with Ruby 2.0?
Sure, and it outputs "test", but I really don't quite see what it has to with what I asked.
It is testing if the core functionality that TextMate.executor relies on is available and works in ruby 2.0.
In Ruby 2.0 (but not in Ruby 1.8.7 or Ruby 1.9.3) we are failing on the "for_fd" line. The question is what to do about that.
Figure out which assumption 2.0 breaks, which is what I was trying to do by having you run the code that tests our core assumptions (creating a pipe and getting the write-end by creating a file descriptor from its integer representation).
Could it be that the TM_ERROR_FD pipe created in executor.rb is
somehow not surviving? Is it being closed too soon somehow?
It does look like it is being closed, though _nothing_ in the user script should close this file descriptor, so the most likely explanation would be, that ruby 2.0 closes all open file descriptors (except 0-2) and _before_ handling at_exit code.
I find this a little weird though, if they are actually doing that - should be possible to test though.
Okay, thanks for the clue; following that up, I was able to discover this: system() and exec() and spawn() and IO.popen() in ruby 2.0 apparently close non-standard file descriptors by default (that is to say, everything except 0, 1, and 2). It may be that this can be prevented by passing :close_others=>false (which can itself be overridden by setting the fd's close-on-exec flag).
We are now well out of my depth. :) But it does seem to me that this has something to do with the case (though I do not know exactly what), since it is raising exactly the sort of red flag of incompatibility that we're looking for.
More info at http://ruby-doc.org/core-2.0/Kernel.html with heavy details particularly under "spawn".
Let me know if there's something you want me to try! Thx - m.
On Mar 8, 2013, at 7:02 PM, Matt Neuburg matt@tidbits.com wrote:
[…] following that up, I was able to discover this: system() and exec() and spawn() and IO.popen() in ruby 2.0 apparently close non-standard file descriptors by default
Woah! That would cetainly explain it, assuming you use ruby 2.0 for the actual RubyMate script.
We generally recommend that you use ruby 1.8.7 for all the TextMate commands (i.e. have /usr/bin first in your PATH) and then set TM_RUBY to whatever ruby you want used for user scripts.
Do you have any reference on ruby 2.0 closing non-standard file descriptors? I only found the “:close_other” option for spawn, but it doesn’t say it is the default behavior.
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 8, 2013, at 7:02 PM, Matt Neuburg matt@tidbits.com wrote:
following that up, I was able to discover this: system() and exec()
and spawn() and IO.popen() in ruby 2.0 apparently close non-standard file descriptors by default
Woah! That would cetainly explain it, assuming you use ruby 2.0 for the actual RubyMate script.
We generally recommend that you use ruby 1.8.7 for all the TextMate commands (i.e. have /usr/bin first in your PATH) and then set TM_RUBY to whatever ruby you want used for user scripts.
I don't think it's reasonable to assume that people will do something so complex. I for one have gone to great trouble to ensure that I am using one version of Ruby globally; it is unlikely that I would deliberately undermine that. Also it is shaky to assume that Ruby 1.8.7 will persist into the future. We are entering an age when Ruby 2 *is* Ruby. That is the future for which we must plan. TextMate's internal scripts must be prepared to deal with whatever version of Ruby they encounter in the real world that surrounds them...!
Do you have any reference on ruby 2.0 closing non-standard file descriptors? I only found the ":close_other" option for spawn, but it doesn't say it is the default behavior.
The key location that got me started is https://github.com/ruby/ruby/blob/v2_0_0_0/NEWS, where we read:
* incompatible changes: * system() and exec() closes non-standard file descriptors (The default of :close_others option is changed to true by default.)
I've seen this expressed elsewhere in a slightly different way:
"Kernel#system and #exec does not inherit non-standard file descriptor by default."
And, as I said, take a look at http://ruby-doc.org/core-2.0/Kernel.html - under "spawn" we are explicitly told that ":close_others is true by default for spawn and IO.popen", and then we are told under "system" and "exec" that their options are the same as for "spawn". Also, in http://www.ruby-doc.org/core-2.0/IO.html, we are told this:
"Ruby sets close-on-exec flags of all file descriptors by default since Ruby 2.0.0. So you don't need to set by yourself. Also, unsetting a close-on-exec flag can cause file descriptor leak if another thread use fork() and exec() (via system() method for example). If you really needs file descriptor inheritance to child process, use spawn()'s argument such as fd=>fd."
Hope this helps - m.
On Mar 9, 2013, at 5:21 PM, Matt Neuburg matt@tidbits.com wrote:
[…] TextMate's internal scripts must be prepared to deal with whatever version of Ruby they encounter in the real world that surrounds them...!
The problem is that ruby is being updated in a way where new versions can’t always run old scripts. It’s simply impractical to ensure that all ruby scripts used by TextMate work with every version of ruby, and for that reason, we only guarantee it works with the one that Apple includes with the OS.
When Apple moves to 2.0, we’ll update the scripts accordingly, until then, we must remain compatible with 1.8.7, but we accept patches when we can remain compatibile with both :)
Do you have any reference on ruby 2.0 closing non-standard file descriptors? […]
[…] as I said, take a look at http://ruby-doc.org/core-2.0/Kernel.html - under "spawn" […]
Tbanks, I see I didn’t read far enough.
Hope this helps
I pushed an update where we explictly call the system ruby for the host script. I don’t see a way we can tell ruby 2.0 we don’t want this new feature without breaking compatibility with 1.8.7.
I didn’t update the unit test runner, as I wanted to be sure this change works (the old way RubyMate was called, is a little weird).
Allan Odgaard mailinglist@textmate.org wrote:
I pushed an update where we explictly call the system ruby for the host script. I don't see a way we can tell ruby 2.0 we don't want this new feature without breaking compatibility with 1.8.7.
OK, I'll boldly try this in my copy of TextMate 1.5 and report back.
Meanwhile I have some immediate thoughts:
What if system ruby *is* Ruby 2.0? This could happen in the future, for all one knows.
Isn't it desirable to be genuinely compatible with both 1.8.7 / 1.9.3 on the one hand, and 2.0 on the other? It is easy to check for this thanks to RUBY_VERSION. (This is a constant defined in Kernel module, hence universally available.) My own code is full of checks of this sort. Ugly but necessary during this transitional phase...
m.
Allan Odgaard mailinglist@textmate.org wrote:
I pushed an update where we explictly call the system ruby for the host script. I don't see a way we can tell ruby 2.0 we don't want this new feature without breaking compatibility with 1.8.7.
This works perfectly. Thank you! When my scripts run inside TextMate under Ruby 2.0, then, when we hit an exception, the exception report is now formatted correctly. Bravo.
An easy test, if anyone is following along, is this script:
puts RUBY_VERSION what
Run that as a Ruby script inside TextMate. First we print the Ruby version, so you know you really are using 2.0. Then we deliberately trigger a NameError by mentioning an undefined local variable. In the RubyMate output, you will see everything nicely formatted, just as under Ruby 1.8.7 and Ruby 1.9.3.
Still mulling over the wisdom of assuming that system ruby is 1.8.7. m.
PS By the way, if anyone is looking for a way to try out Ruby 2.0 on Mac, I recommend rbenv. So easy. So harmless.
On Mar 11, 2013, at 8:16 PM, Matt Neuburg matt@tidbits.com wrote:
[…] What if system ruby *is* Ruby 2.0? This could happen in the future, for all one knows.
Yes, and python could become py3k, bash could become zsh, etc. Let’s solve the issue when it happens :)
If Apple were to include ruby 2.0 though, I’m almost certain they would do so by introducing it as a ruby20 binary (or similarly named), since it’s not a backwards compatible replacement.
Isn't it desirable to be genuinely compatible with both 1.8.7 / 1.9.3 on the one hand, and 2.0 on the other?
If that compatibility came for free, sure. Since it is not free, the real question is, how many resources should we devote on maintaining compatibility with three versionf of ruby (1.9 can’t run all 1.8 code, and 2.0 can’t run all 1.9 code)?
It is easy to check for this thanks to RUBY_VERSION.
It’s also easy to just use ruby 1.8 to run the scripts that were written for ruby 1.8.
And while it’s easy to test for RUBY_VERSION, it’s not easy to actually write the code, because you need to be familiar with the subtle details of all 3 major versions of ruby and test on all 3 to be sure.
On Mar 11, 2013, at 8:36 PM, Matt Neuburg matt@tidbits.com wrote:
[…] if anyone is looking for a way to try out Ruby 2.0 on Mac, I recommend rbenv. So easy. So harmless.
And remember, set TM_RUBY to your desired version rather than change PATH to have your custom install eclipse the one in /usr/bin :)
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 11, 2013, at 8:36 PM, Matt Neuburg matt@tidbits.com wrote:
if anyone is looking for a way to try out Ruby 2.0 on Mac, I recommend rbenv. So easy. So harmless.
And remember, set TM_RUBY to your desired version rather than change PATH to have your custom install eclipse the one in /usr/bin :)
I've never actually figured out what you mean by that. The way I do this *is* by changing TextMate's PATH:
/Users/mattleopard/.rbenv/bin:/Users/mattleopard/.rbenv/shims:/usr/bin:/ bin:/usr/sbin:/sbin
This causes us to see both rbenv and the ruby that rbenv has currently set. Thus I can nimbly switch from one ruby to another (using rbenv) and TextMate will use whatever I've switched to. I believe this is what *every* rbenv user does in order to make TextMate work. m.
On Mar 12, 2013, at 8:42 PM, Matt Neuburg matt@tidbits.com wrote:
[…] set TM_RUBY to your desired version rather than change PATH to have your custom install eclipse the one in /usr/bin :)
I've never actually figured out what you mean by that. The way I do this *is* by changing TextMate's PATH […]
You can set "TM_RUBY=$HOME/.rbenv/versions/1.9.3-p327" or similar.
That said, we’re planning to update all ‘ruby’ calls to use an absolute path to make it more robust.
Allan Odgaard mailinglist@textmate.org wrote:
On Mar 12, 2013, at 8:42 PM, Matt Neuburg matt@tidbits.com wrote:
[…] set TM_RUBY to your desired version rather than change PATH to have
your custom install eclipse the one in /usr/bin :)
I've never actually figured out what you mean by that. The way I do this *is* by changing TextMate's PATH […]
You can set "TM_RUBY=$HOME/.rbenv/versions/1.9.3-p327" or similar.
The whole point of rbenv (or any similar ruby version management system) is that I give a command at the command line, like this:
$ rbenv global 2.0.0-p0
And presto, my entire system sees that ruby as its ruby, along with all concomitants (the corresponding rdoc command, gem command, actual gems, library, etc.).
The problem is that TextMate doesn't get the memo. The reason is that TextMate doesn't pass through my .bash_profile, so it doesn't have $HOME/.rbenv/bin in its path and doesn't do eval "$(rbenv init -)". To solve this problem, I have modified TextMate's PATH so that it starts like this:
/Users/mattleopard/.rbenv/bin:/Users/mattleopard/.rbenv/shims: ...
The first is so that the "rbenv" command itself is visible. The second is so that rbenv's "ruby", "rdoc", "gem" and other shims are visible. That's the genius of rbenv - those commands are actually shims, simple scripts that turn to rbenv itself to exec the real command.
So, the point is, I'm already using the rbenv command at the command line to switch, nimbly, between ruby versions. I do this many times a day, testing a script under different ruby configurations. There is not a snowball's chance that I'm *also* going to give some *explicit* command pointing TextMate's TM_RUBY at a specific ruby. That would not be nimble at all! TextMate's job is to follow the lead I am *already* giving when I set my ruby version globally thru rbenv. My adjustment to the PATH is how I get TextMate to follow that lead.
As I've said before, as far as I know, this is what *every* TextMate user does (if they want to be nimble with Ruby versions).
So, I'm not sure where TextMate is headed with this, but I hope it's in the direction of playing even *more* nicely with rbenv and my choice of global ruby version - not less nicely. I have a lot of functionality built upon this; obviously I don't want it to break. m.
On 23 Mar 2013, at 22:34, Matt Neuburg wrote:
I have modified TextMate's PATH so that it starts like this:
/Users/mattleopard/.rbenv/bin:/Users/mattleopard/.rbenv/shims: ...
The first is so that the "rbenv" command itself is visible. The second is so that rbenv's "ruby", "rdoc", "gem" and other shims are visible.
OK, so it sounds like this might work (be aware that only 2.0 will expand $HOME and $PATH in your variable settings):
TM_RUBY = "$HOME/.rbenv/shims/ruby" TM_RI = "$HOME/.rbenv/shims/ri" PATH = "$PATH:$HOME/.rbenv/bin"
I don’t know if the last line is required (for the shims to function), but by appending to PATH we don’t eclipse any of the standard tools, so it does no harm for TextMate.
So, I'm not sure where TextMate is headed with this, but I hope it's in the direction of playing even *more* nicely with rbenv and my choice of global ruby version - not less nicely. I have a lot of functionality built upon this; obviously I don't want it to break. m.
As mentioned in my previous reply, we’ll (hopefully soon) start to hardcode the /path/to/ruby to make it robust against user alterations.
On 2013-03-24 10:32, Allan Odgaard wrote:
As mentioned in my previous reply, we’ll (hopefully soon) start to hardcode the /path/to/ruby to make it robust against user alterations.
I hope we'll still be able to run Ruby scripts via RVM and similar tools when pressing Cmd+R.
-
Allan Odgaard -
Jacob Carlborg -
matt＠tidbits.com