I think you're right. I cant remember what editor I saw which uses this... maybe Atom or Sublime, and also vim with a plugin.

Probably you're talking about this plugin: https://github.com/abe33/atom-pigments

AFAIK, there's no way to implement this without modifying TextMate source code.