Hi,

OK. Step by step.

First we have to test the Rdaemon whether it really works stable. Thus, please test it against large R projects! [But be careful, do not use critical data]

The only known pitfalls are: -fix() and edit() don't work (it only opens a TM window) -a<-readline("test") works but one has to enter the value in a separate line (up to now) -quartz() only works if one has installed the CarbonEL package

Installation: -install R daemon.tmbundle -open the gear menu (CTRL+ESC) for R daemon and invoke the last item 'Install Rdaemon'

That's it.

General stuff: -R daemon listens at $HOME/Rdaemon/r_in -the entire session is written into $HOME/Rdaemon/r_out -the result of selected commands or if you run your commands in the background will be written into $HOME/Rdaemon/r_res -errors/messages will be written into $HOME/Rdaemon/r_err -there is a persistent history for Rdaemon (OPT+ARROWUP/DOWN or a list CTRL+OPT+ARROWUP) -working directory for the Rdaemon is $HOME/Rdaemon/ -r_out, r_err will be overwritten if you start Rdaemon

Workflow: [a dilettante made screencast is here http://www.bibiko.de/ Rsession2.mov (sorry 20MB) ] -open a new document and set language to 'R console' -type 'start' and press TAB to start the daemon -wait a bit until '> ' occurs -type whatever you want and execute the line or selection by pressing NUMPAD+ENTER -to quit type 'q("yes")' or 'q("no")' + NUMPAD+ENTER (.Rdata will be written into $HOME/Rdaemon; if you don't quit R but you quit TM, R daemon runs further, so you can start TM again and continue your stuff) -there is a basic TM project called Rsession available which I recommend to use to simplify some things

In addition to that I tried to write a kind of a GUI around it. You can invoke the R commander with CTRL+OPT+APPLE+R. This R commander is only a PROTOTYPE!! The idea is to use an ordinary HTML output window of a tmCommand to remote the entire Rdaemon session. If the GUI works in the daily life a very nice side-effect could arise. You have a quite simple to modify GUI. It's written in basic HTML/JavaScript plus a bit shell commands. In other words everyone could set up her/his frequent used R commands in order to save time. AND the possibilities of that GUI are almost endless if one has time to write code ;) [and I have dozens of ideas] BTW, the intent of this GUI is NOT to replace the wonderful R.app!

I played with Rdaemon and the GUI. I ran several scripts etc. and it turned out that if one has understood the workflow once one can do a lot of stuff faster and one is more flexible (at least up to now ;).

OK. But we have to attest firstly whether Rdaemon runs stable!

I would be very appreciate if someone can test it. Any hint - illogical behaviour, crashes, handling stuff, speed, etc. - is highly welcomed.

--Hans

Hi Hans,

Thanks for the update. You're improving the daemon quite a lot faster than I can try things out, but I've found a few issues:

- You seem to be using ⌘⎋ for autocompletion, but that conflicts with Front Row - Trying to access help by typing '?print' or 'help("print")', for example, blocks TM waiting for a (very CPU-intensive) perl process that appears to run indefinitely - I know I should be using ⌃H but it would be good if 'help' at least failed cleanly - browser() doesn't seem to work properly, so debug(function) is unusable

With regard to the RAM drive, my feeling is that it's best not to block out a chunk of memory like that, particularly since older systems on which autocomplete will be slower are also likely to be the ones that can least spare the RAM. Personally, I find it quick enough as it is.

HTH, Jon

On 20/12/2007, Hans-Joerg Bibiko bibiko@eva.mpg.de wrote:

...

Hi. I believe, now it works perfectly in some sense. I fixed the synchronisation by using the expect approach, meaning R must return something after doing the task, such as '> ', '+ ' or ': '. For that reason I had to merge stdout and stderr. Thus the error/warning messages appear in the document, but one can press 'undo' easily. This means further that EACH prompt for a readline command MUST be end of ': '! Otherwise TM blocks (press APPLE+. for cancelling). For all internal R commands is that the case.

I tested it and I provoked fatal errors. Even for fatal errors R asked me to quit (with the option to save the current workspace or not!). I ran big scripts, I ran R inside of R, etc. The daemon interacted stable.

I fixed dozens of tiny things, I added a package manager (for loading/ detaching), a real workspace browser with edit/delete function. edit(), fix() work now BUT these commands must be executed in the background. Otherwise TM will freeze (APPLE+. for cancelling) because I set the editor to 'mate -w'!

I discarded the idea of using an HTML window as a kind of GUI because there're some tiny hitches. I went the TM way by overloading key combinations, e.g. OPT+APPLE+G lists all relevant commands for 'Graphics' etc.

Then my next idea was to speed up that daemon. My approach is to spawn R à la R --encoding=UTF-8 --TMRdaemon 2&> r_out

Fine. That works brilliant. But if I create a ram drive of 80MB and pipe R's output to /tmp/TMRramdisk1/r_out, it's speed up the entire Rdaemon extraordinarily. Auto-completion, inserting of command templates, the output behaviour work almost instantaneously! I never saw that before. But there're two tricky side-effects:

1. The ram drive is limited to 80MB. It should be enough, and it'll

cleaned up by each starting of Rdaemon. 2) Detaching a ram drive. If one uses hdiutil detach, well, sometimes it works, sometimes not. It unmounts it all the time but it couldn't eject it, caused by the issue that the device is busy(?). I only can guess that QuickSilver, spotlight, anti-virus programs have a handle to that device. OK. If one restarts Mac that ram drive is gone. If one logs out and in the ram drive will be still there. If I start the Rdaemon the script is checking if that ram drive is available, if not it will create it. Thus one can work as usual, one 'only' looses 80MB of ram.

So, my BIG question is whether we should use a ram drive. By my opinion, yes, because it is so fast and it makes fun. 80MB is not too much.

Any comments?

Cheers, --Hans

---

For new threads USE THIS: textmate@lists.macromates.com (threading gets destroyed and the universe will collapse if you don't) http://lists.macromates.com/mailman/listinfo/textmate

Hi Jorg,

I just started using this and it look pretty sweet so far (though admittedly I haven't gone far ;) ). The only problem I am seeing so far is with plots, the first time they are generated. If the plot window is closed, and I run something like "plot(1:10)", sometimes I get:

326:331: execution error: System Events got an error: NSReceiverEvaluationScriptError: 4 (1) or sometimes 331:336: execution error: System Events got an error: NSReceiverEvaluationScriptError: 4 (1)

Though actually I cannot reproduce this now ;).

Anyway, very nice work! I've been copying and pasting for way too long ;).

I'll let you know if I find any other problems, I tend to use R quite a lot these days.

The only thing that would worry me about the RAM drive idea is the question of handling large data sets. Probably I am misunderstanding how it works, but what happens if you load some data that would take more than 80M?

Best, Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Dec 20, 2007, at 10:42 AM, Hans-Joerg Bibiko wrote:

On 20 Dec 2007, at 15:10, Jon Clayden wrote:

...

Thanks for the update. You're improving the daemon quite a lot faster than I can try things out, but I've found a few issues:

• You seem to be using ⌘⎋ for autocompletion, but that

conflicts with Front Row

OK. key bindings is the last issue

...

• Trying to access help by typing '?print' or 'help("print")', for

example, blocks TM waiting for a (very CPU-intensive) perl process that appears to run indefinitely - I know I should be using ⌃H but it would be good if 'help' at least failed cleanly

Is already fixed. As default ?topic shows the HTML page in the default browser, but you also can change this to man pages (which will be cleaned of course)

...

• browser() doesn't seem to work properly, so debug(function) is

unusable

I already changed the entire process handling. Thus browser(), debug () work but I will fine-tune it a bit because now the prompt will be like:

...

myf <- function(x) {for(i in 1:10){print(i);a <- i*2;browser()}} myf()

[1] 1 Called from: myf() Browse[1]> |

'|' = caret

If you want to enter something you have to press ENTER first then you can continue typing debug commands. But I believe I will get rid of it ;)

...

With regard to the RAM drive, my feeling is that it's best not to block out a chunk of memory like that, particularly since older systems on which autocomplete will be slower are also likely to be the ones that can least spare the RAM. Personally, I find it quick enough as it is.

Well, I believe I will manage this by using customisable start options. By doing so the user can choose whether Rdaemon should run on a ram drive or not, including the chance to set the ram size.

Please be a bit patient, but it is really amazing to see how fast Rdaemon is running on a ram drive. It makes really really fun and together with TM's functionality and flexibility it's almost unbeatable ;)

--Hans

On Dec 20, 2007, at 1:04 PM, Hans-Jörg Bibiko wrote:

On 20.12.2007, at 17:28, Charilaos Skiadas wrote:

...

I just started using this and it look pretty sweet so far (though admittedly I haven't gone far ;) ). The only problem I am seeing so far is with plots, the first time they are generated. If the plot window is closed, and I run something like "plot(1:10)", sometimes I get:

326:331: execution error: System Events got an error: NSReceiverEvaluationScriptError: 4 (1) or sometimes 331:336: execution error: System Events got an error: NSReceiverEvaluationScriptError: 4 (1)

Did you open a quartz device or x11? The error message tends to be sent from quartz. Did you installed CarbonEL package? Anyway I will try to reproduce it.

No I did not open a quartz device by myself, so far I got this error from three different places: 1. When for the first time a quartz window opened as a consequence of a call to "plot", and a couple more times later 2. When A window opened by a call to a lattice command, namely histrogram. Strangely enough, once that happened once, from that point on neither plot or histogram gave me another error 3. Whenever I try to open a help page, unless that item is already open in help (help pages open within R.app for me).

...

I'll let you know if I find any other problems, I tend to use R quite a lot these days.

I hope that I can post the latest version tomorrow, so you have something better to test ;)

...

The only thing that would worry me about the RAM drive idea is the question of handling large data sets. Probably I am misunderstanding how it works, but what happens if you load some data that would take more than 80M?

No. There is no limit for data size. The only purpose of the ram drive is store the console output coming from R, in other words what you writing and printing out while a session within the console.

I see. Then it's probably a good idea, though 80M might be too much, I dunno. Why did you set it to 80M? Perhaps the size can be customizable as well.

Btw, a thing to consider adding is to make it so that, when on the last line, using up/down arrows moves you among the history items, like the R.app GUI does. This action has been built too much into my muscle memory by now.

Btw, I had some problems killing the daemon. I tried kill+tab, which resulted in the message "Rdaemon is not running!". Then trying start +tab resulted in "Rdaemon is already running". Indeed the R daemon seems to work throughout this process. ps gives me:

charilaos-skiadas-ibook-g4-3:~/Desktop/latexdiff haris$ ps -auxw|grep Rdaemon haris 7999 0.2 -0.0 28120 932 ?? S 11:19AM 0:12.59 ruby /Users/haris/Rdaemon/daemon/Rdaemon.rb haris 8001 0.0 -0.0 27808 700 p2 Ss+ 11:19AM 0:00.00 sh -c R --encoding=UTF-8 --TMRdaemon 2> /Users/haris/Rdaemon/r

Btw, when the daemon first started, it started X11. Is that necessary? I usually don't have X11 running.

--Hans

Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Dec 20, 2007, at 2:11 PM, Hans-Jörg Bibiko wrote:

On 20.12.2007, at 19:43, Charilaos Skiadas wrote:

...

On Dec 20, 2007, at 1:04 PM, Hans-Jörg Bibiko wrote:

No I did not open a quartz device by myself, so far I got this error from three different places:

1. When for the first time a quartz window opened as a consequence

of a call to "plot", and a couple more times later 2. When A window opened by a call to a lattice command, namely histrogram. Strangely enough, once that happened once, from that point on neither plot or histogram gave me another error 3. Whenever I try to open a help page, unless that item is already open in help (help pages open within R.app for me).

Wait. I look for 'System Events got an error: NSReceiverEvaluationScriptError: 4' at the internet . All pages refer to an applescript error. But the Rdaemon do not need applescript, actually?? Did you run R.app at the same time?

Yes of course, if my computer is on, then R.app is running ;) Ok, I just figured out what I was doing wrong, stupid of me. An R Console command was used instead because I was just pressing return. I was thinking this would be the trigger for the Rdaemon command. So it wasn't your code that was running at all, it was my old crappy code from the RConsole/Applescript days. Using enter instead of return, i.e. your code, things work just fine. Graphs open in x11 windows. Help is still a problem though, TM just stalls.

Why not set return to be the trigger for the execute command, instead of enter?

killing is also still a problem.

...

I see. Then it's probably a good idea, though 80M might be too much, I dunno. Why did you set it to 80M? Perhaps the size can be customizable as well.

I'm just working on that ;)

...

Btw, a thing to consider adding is to make it so that, when on the last line, using up/down arrows moves you among the history items, like the R.app GUI does. This action has been built too much into my muscle memory by now.

Hmm... I will sleep about it whether there is an easy to implement this.

The last line is the only one not ending with a newline character, so it should be doable to detect that.

...

Btw, I had some problems killing the daemon. I tried kill+tab, which resulted in the message "Rdaemon is not running!". Then trying start+tab resulted in "Rdaemon is already running". Indeed the R daemon seems to work throughout this process. ps gives me:

It's also already fixed. I changed the command line to call R, and I forgot to change the grep regexp.

...

Btw, when the daemon first started, it started X11. Is that necessary? I usually don't have X11 running.

I did this for safety reasons. chooseCRANmirror() or other stuff opens a tk menu. I already set options(menu.graphics=F) while starting but sometimes I see such a tk menu ? Maybe it's caused by the issue that X11 runs? I'll check this

--Hans

Haris Skiadas Department of Mathematics and Computer Science Hanover College

On Dec 20, 2007, at 2:50 PM, Hans-Jörg Bibiko wrote:

...

Why not set return to be the trigger for the execute command, instead of enter?

Good question. For me, it's quite handy because I use the R console as text document AND R console, but I guess everyone can change this easliy.

Fair enough. I personally would prefer having enter force a newline and return be execute. But I guess that's a change I can make locally.

...

...

...

Btw, a thing to consider adding is to make it so that, when on the last line, using up/down arrows moves you among the history items, like the R.app GUI does. This action has been built too much into my muscle memory by now.

Hmm... I will sleep about it whether there is an easy to implement this.

The last line is the only one not ending with a newline character, so it should be doable to detect that.

I guess this I only can do with TMTOOLS. I bind the key ARROW-UP/DOWN to tmcommand; it will detect whether there's a \n or not; if not then invoke the history stuff; BUT now it comes the big question how to send the key event MoveUp/Down to TM if there's a \n ?? Any ideas?

Frankly I have not looked at TMTOOLS, at all, so I probably can't help much there. I was thinking of perhaps matching this in the grammar level. I can sort of do it with a syntax like:

{ scopeName = 'source.r-console'; fileTypes = ( ); patterns = ( { name = 'source.r.embedded.r-console'; begin = '^> '; end = '\n|\z'; beginCaptures = { 0 = { name = 'punctuation.section.embedded.r- console'; }; }; patterns = ( { name = 'variable.testing'; match = '.*[^\n]\z'; }, { include = 'source.r'; }, ); }, ); }

But then it is hard to have the last line also be syntax colored according to the R syntax. But my grammar-foo is quite rusty, to say the least.

--HANS

Haris Skiadas Department of Mathematics and Computer Science Hanover College

Hi,

here comes the next improved version of the Rdaemon.

Installation: -install R daemon.tmbundle as usual -open the gear menu for R daemon and execute 'Install Rdaemon' [make sure that the folder ~/Rdaemon doesn't exist in beforehand]

Usage: -open a new document and set the language to 'R console (Rdaemon)' -press OPT+APPLE+C to show the general control menu -choose 'Start Rdaemon' -type a command and press RETURN or ENTER

[look at the menu structure of the Rdaemon bundle in order to see what's possible]

Key binding: If you are at a command line (with beginning > + : or Browse) RETURN executes the command. The same for ARROW UP/DOWN for history. If you are inside of a text or whatever RETURN + ARROW UP/DOWN are set to the default behaviour. To use the default behaviour if your inside of a command line use APPLE+RETURN, or APPLE+ARROW UP/DOWN

ENTER always executes the current command.

Auto-completion is now bound to CTRL+.

Notes: 1) I improved the interaction between TM and R in such a way, that TM will always wait for a response from R, meaning TM expects a '> ', '+ ', or ': '. Otherwise TM will freeze (cancelling by pressing APPLE+.).

1a) To make sure that readline commands work you have to append ': ' to each prompt! Example:

a <- readline("value for x: ")

value for x:

|

| = caret. Enter now the value for a and press RETURN

Otherwise TM will freeze. For all internal prompts I checked is that the case.

2) edit() and fix() work if you execute these commands as background task (SHIFT+ENTER). Otherwise TM will freeeze because R calls 'mate -w'! If you want to edit an object use the 'Show Workspace' (OPT+APPLE+W) command and press the 'edit' button.

3) I wrote a Package Manager for loading/detaching of installed packages (OPT+APPLE+P). Use the checkboxes for that.

[The next item at the todo list is the write an 'Install Package' GUI]

4) quartz() ONLY works if you installed the 'CarbonEL' package!!

5) As default ?topic will open the HTML page in the default browser. You can change this in options().

6) Rdaemon comes with its own language grammar.

7) RAM drive: r_out (the entire console data) are stored in ~/Rdaemon/r_out. If you have a bit free RAM you can start Rdaemon in the RAM drive mode, meaning r_out (and some temp files) will now stored at a RAM drive which increase the speed enormously. To set up a RAM drive make the following: -go to TM's preference pane and choose Advanced > Shell variables -create a new variable called 'TM_RdaemonRAMDRIVE' and set it to '1' -start Rdaemon as usual -in addition you can specify the RAM size with the shell variable 'TM_RdaemonRAMSIZE'; the default is 50MB -while running Rdaemon with a RAM drive the daemon will check the free space and outputs a message if the free space is less than 3% for safety reasons. Then you can save the workspace; restart Rdaemon; and load the workspace

-if you set TM_RdaemonRAMDRIVE to '0' or delete it Rdaemon runs using the hard disk -! unfortunately there is no way to detach the ram drive; the only way is to restart your Mac ! (I'm working on it)

8) CTRL+H opens a help menu for the current word. Sometimes if you click 'fast' at a link inside of the HTML page TM doesn't follow that link. Thus click 'slower' and then it works ;)

9) For some GUI commands I make usage of the sink() command. Normally it works fine. But if you type e.g. '2+4' press RETURN and you don't see the reslut (only '> ') it could be that sink was not reset. To reset sink invoke OPT+APPLE+M and execute 'Reset Output' once or twice until you the correct output. As I mentioned this only occurs if you develop stuff.

So, please test it and give feedback. I would be very appreciated to receive any kinds of comments on it.

Have fun (hopefully)

--Hans

Hi,

I fine-tuned Rdaemon a bit, sped it up, and I added some new tools.

Changes: -If the progress bar appears one sees the last line of the console. I.e. if one executes install.package('foo') one sees at least what R is doing.

-If one presses '(' TM not only inserts '()' but TM also displays a tool tip showing the command usage.

-If one presses '$' TM lists all available attributes to that object for inserting.

[Now my favourite. I satisfied my longed for wish ;)] -CTRL+, This key combo displays all named parameters of the current function. One can select one parameter and it inserts it as snippet. If the parameter has a default only the default value will be inserted as snippet. This ALSO works for commands with nested functions! Up to now it only works inside of the current line.

At least for me it's often the case that I see the command usage as a tool tip but while writing I forget the correct spelling of parameters. In R.app you see the command usage in the status bar but if you're typing nested functions, ...

See demo http://www.bibiko.de/TMRdaemonTools.mov

I also tried out the iplots package and it also works very good.

I thought about how to save a plot. One solution is to use Preview (OPT+APPLE+G) and save it as PNG,TIFF,PDF, etc. Comments?

Have fun and Happy New Year

--Hans

On 3 Jan 2008, at 23:50, Peter Cowan wrote:

This is wonderful, your efforts are greatly appreciated. Perhaps it is time to replace the R console code in the main R package with these commands?

Coming soon.

The code completion would be useful when editing normal R scripts as well, is that easily achieved?

Is under construction. Basically it works pretty good without calling R – meaning all these commands work on R's help file system.

A few very minor comments.

1. The insert argument and parameter function inserts in the

following fashion "n=1". However, the R coding standard[1], suggests "n = 1" (with spaces). I suspect that many will want it the way it is, and it's not hard to change the command, but perhaps the default should be changed.

I agree. It's fixed.

2. Why does the bundle include a style for the prompts? Why not

scope this and have it handled by the chosen theme? For some themes the default isn't very attractive to my eyes.

Actually I didn't find a adequate scope keyword for that more or less new stuff in TM. But I agree, I will change the prompt to keyword.*** and the browser prompt to comment.***

3. The bundle I have installed has two "Start X11" and two "Open

Rsession Project" commands, what is the difference between these?

I overloaded key combos. I.e. e.g. for X11 it belongs to 'general commands' and to 'graphics'.

[1] To the extent that it exists: http://cran.r-project.org/doc/manuals/R-exts.html#Tidying-R-code

This was a good hint. I try write a Tidy-up command for R code.

--Hans