[TxMt] Re: Regression tests for grammars?

[Michael Sheets]

On Jul 11, 2014, at 4:27 AM, Stefan Daschek <stefan at daschek.net> wrote:

> From my - admittedly somewhat limited - experience with bundle grammars I got the feeling that it’s often very hard to improve or fix a grammar without accidentally breaking some other parts of it.
> 
> Has anyone already experimented with regression tests for bundle grammars?

The biggest issue with tests for grammars is simply writing tests that are worthwhile. The errors you typically see are the exceptional cases you wouldn’t have thought to test for. But it can prevent the same error from happening twice.

The second issue is that there is no ‘perfect’ way to scope a portion of code and thusly the common manner of writing tests would break every time you made even the slightest change to the grammar. To be something people would use it needs to be relatively painless.

I’ve put a lot of thought into how to solve this for grammars and my current best thoughts are:

A two part test with a string of text to be matched and then a series of assertions at various points in the string. An example:

	"foo\"bar"
	
	1:6 – B:constant.character.escape
	1:8 – string

The string would be parsed and then the matches provide a caret position and a scope to match against. Here we use a B: scope to ensure we are inside an escape and then after that point ensure the string didn’t end early. The scopes are very intentionally as vague as possible to prevent minor changes in scopes from invalidating the test. Here we are only worried about the string and the escape so it’s all we test for.