From my - admittedly somewhat limited - experience with bundle grammars I got the feeling that it’s often very hard to improve or fix a grammar without accidentally breaking some other parts of it.
Has anyone already experimented with regression tests for bundle grammars?
I think to make this possible it would be necessary to somehow access the “scope structure” of a parsed file, maybe yielding a data structure like this:
[ { start: 0, end: 123, scope: "source.sass" }, { start: 7, end: 12, scope: "string.quoted.double.sass" }, … ]
Then a grammar could include both a “test source file” and its known-good “scope structure”. After changing the grammar one could (semi-)automatically regenerate the scope structure data and compare it to the original to (hopefully) catch any regressions.
Any thoughts?
Stefan
On Jul 11, 2014, at 4:27 AM, Stefan Daschek stefan@daschek.net wrote:
From my - admittedly somewhat limited - experience with bundle grammars I got the feeling that it’s often very hard to improve or fix a grammar without accidentally breaking some other parts of it.
Has anyone already experimented with regression tests for bundle grammars?
The biggest issue with tests for grammars is simply writing tests that are worthwhile. The errors you typically see are the exceptional cases you wouldn’t have thought to test for. But it can prevent the same error from happening twice.
The second issue is that there is no ‘perfect’ way to scope a portion of code and thusly the common manner of writing tests would break every time you made even the slightest change to the grammar. To be something people would use it needs to be relatively painless.
I’ve put a lot of thought into how to solve this for grammars and my current best thoughts are:
A two part test with a string of text to be matched and then a series of assertions at various points in the string. An example:
"foo"bar" 1:6 – B:constant.character.escape 1:8 – string
The string would be parsed and then the matches provide a caret position and a scope to match against. Here we use a B: scope to ensure we are inside an escape and then after that point ensure the string didn’t end early. The scopes are very intentionally as vague as possible to prevent minor changes in scopes from invalidating the test. Here we are only worried about the string and the escape so it’s all we test for.
Hello all, I love this and it could be used also to validate/test theme coverage. What do we have to do to get something like this created so bundle authors can use it?
Take care,
Jeremy Whitlock (@whitlockjc on Twitter) http://www.thoughtspark.org
On Mon, Jul 14, 2014 at 2:03 AM, Michael Sheets mummer@whitefalls.org wrote:
On Jul 11, 2014, at 4:27 AM, Stefan Daschek stefan@daschek.net wrote:
From my - admittedly somewhat limited - experience with bundle grammars
I got the feeling that it’s often very hard to improve or fix a grammar without accidentally breaking some other parts of it.
Has anyone already experimented with regression tests for bundle
grammars?
The biggest issue with tests for grammars is simply writing tests that are worthwhile. The errors you typically see are the exceptional cases you wouldn’t have thought to test for. But it can prevent the same error from happening twice.
The second issue is that there is no ‘perfect’ way to scope a portion of code and thusly the common manner of writing tests would break every time you made even the slightest change to the grammar. To be something people would use it needs to be relatively painless.
I’ve put a lot of thought into how to solve this for grammars and my current best thoughts are:
A two part test with a string of text to be matched and then a series of assertions at various points in the string. An example:
"foo\"bar" 1:6 – B:constant.character.escape 1:8 – stringThe string would be parsed and then the matches provide a caret position and a scope to match against. Here we use a B: scope to ensure we are inside an escape and then after that point ensure the string didn’t end early. The scopes are very intentionally as vague as possible to prevent minor changes in scopes from invalidating the test. Here we are only worried about the string and the escape so it’s all we test for.
textmate mailing list textmate@lists.macromates.com http://lists.macromates.com/listinfo/textmate
-
Jeremy Whitlock -
Michael Sheets -
Stefan Daschek