As we've discussed before, the current compiler selection logic (in the Typeset & View command) is a bit of a mess. Below is a suggestion for how it might be rationalised. Once we've agreed on a plan, I'm happy to implement it.
One complication is that it's sometimes useful to have compilation under the control of a custom script – either one that you've written for your specific project, or a generic compilation tool such as latexmk. The way that TeXShop deals with this is to have special program-designators "mytex" and "mylatex", which signify that a user-specified compilation script should be used. I propose that we adopt this mechanism.
The first task is to decide on the compilation route to use. This will be one of: tex, latex, pdftex, pdflatex, xetex, xelatex, mytex, mylatex. I propose trying the following things, in order:
1. Use the %!TEX TS-program specification in the source file.
2. Use the %!TEX TS-program specification in the master file.
3. Use the value of the $TM_LATEX_PROGRAM variable.
4. Sniff for \usepackage commands that suggest (DVI)latex/xelatex should be used.
5. Default to pdflatex.
If the resulting value is mytex/mylatex, then:
1. Use the script specified in the variable $TM_LATEX_MYTEX or $TM_LATEX_MYLATEX,
2. Use the script specified in the TeXShop configuration, if any: