No way to scope a grammar capture

[dead-claudia]

Description

There is no way to scope a grammar capture for specific grammar rules, leading to very odd, buggy-looking syntax highlights.

Steps to Reproduce

1. Set the language to any that has a rule that features both of the following. language-gfm and language-javascript both fulfill this requirement in their grammars.

• A begin and end capture
• A pattern that delegates to another language that also has a rule containing both a begin and end capture. (It doesn't have to delegate, though.)

2. Create something that matches that rule. For GFM, a code block tagged with css would suffice, and for JS, a tagged template starting with html would work.

3. Within that block, type something that would match the begin half of any rule in the other language that includes both a begin and end. Do not include the end. For GFM or JS, just start a comment in CSS or HTML, respectively.

4. Observe very odd syntax highlighting. It looks like a bug because it can't be contained, and repros 100% of the time.

If you followed the above repro, the code might look like this pre-syntax highlighting:

• GFM -> CSS

  ```css
  
  /* This is an unterminated comment
  
  ```
  
  
  
  Some more text...
  

• JS -> HTML

  html`
  
  <!-- This is an unterminated comment
  
  `;
  
  
  
  someOtherCall();
  

Here's a few screenshots of other code.

language-gfm -> language-javascript:

language-javascript -> language-html

Versions

OS: Ubuntu GNOME 14.04 (although it's been repro'd in Windows by others)
Atom: v1.12.5 (been going on for quite a while)

Additional Information

Initially reported in atom/language-gfm#171, but I filed it here because it's language-agnostic.

[winstliu]

Thought about this for a while and I think I have a possible solution: adding a new key, something like alwaysMatchEndPattern which is false by default. When true, it will always match the end pattern, even if it would be otherwise matched by another pattern.

It's ironic that most of the time, we complain about not having enough context when developing grammars, but in this case, we're hit by the problem of knowing too much context.

[dead-claudia]

@50Wliu

Thought about this for a while and I think I have a possible solution: adding a new key, something like allowsMatchEndPattern which is false by default. When true, it will always match the end pattern, even if it would be otherwise matched by another pattern.

SGTM. Is my understanding correct in that this would be executed before the child grammar is run, so the child grammar can be run through just a predetermined slice?

It's ironic that most of the time, we complain about not having enough context when developing grammars, but in this case, we're hit by the problem of knowing too much context.

Yeah...really unusual. 😆

[winstliu]

Yeah, at least I hope so. I haven't looked at the implementation to see if it can be done reasonably though.

[burodepeper]

I am curious if this issue isn't better specified as issues with the offending grammars.

This issue can also be reproduced using a match in the delegated grammar, instead of a begin and end, see: burodepeper/language-markdown#77 (comment) (this is similar to the optional parameter issue reported with JavaScript)

The offending grammar in this case is language-yaml which has the following rule (https://github.com/atom/language-yaml/blob/master/grammars/yaml.cson#L299-L302) which matches, well, unquoted strings. I am not too familiar with YAML, and there is probably a use for it, but it seems like a tricky thing altogether if you ever want to be able to 'escape' the YAML grammar again.

I can understand the replications with comments, and while they serve their purpose to showcase the issue, I think they're valid situations, because they do what they're supposed to do; the broken syntax highlighting shows that there's an error in your code, and above all, something that's easily fixable. Syntax highlighting is supposed to help you with this.

The issue with the optional parameter in JavaScript embedded in Markdown is a lot worse. So is the issue with YAML. It is valid code that appears to have errors. And I might be naive, especially since there are most likely a lot more of these issues, but I think they have to be fixed in their respective grammars.

[winstliu]

the broken syntax highlighting shows that there's an error in your code, and above all, something that's easily fixable.

The problem is that it isn't invalid code.

<script>console.log('hi!'); //</script>

is perfectly valid HTML, for example. HTML will always end the script at a closing tag, even if it would otherwise be commented by JavaScript.

And for Markdown...

```js
console.log('hi!');
/*
```

is also valid, because GFM always ends at the triple backticks, much like HTML.

[burodepeper]

@50Wliu You are probably right, and now I understand your alwaysMatchEndPattern solution. Thanks for clearing that up.