JavaScriptNext language doesn't work

[bottleofwater]

I've tried feeding the following file into FirstMate, but unfortunately it fails with the following message (one for each line in the source code):

Popping rule because it loops at column 0 of line '' [ { rule:
     Rule {
       grammar: [Object],
       registry: [Object],
       scopeName: 'source.js',
       contentScopeName: undefined,
       endPattern: undefined,
       applyEndPatternLast: undefined,
       patterns: [Object],
       scannersByBaseGrammarName: [Object],
       createEndPattern: null,
       anchorPosition: -1 },
    scopeName: 'source.js',
    contentScopeName: undefined } ]

What happens? Is there an incompatibility between FirstMate and some grammars?

[thomasjo]

first-mate doesn't support .tmLanguage files, so in case that's what you're actually doing, you'll need to convert it to a CSON/JSON file first. Easiest approach is likely to use apm, but if you want to do it programmatically, parse the .tmLanguage file with plist and then either pass that directly to first-mate, or save it to disk first as CSON/JSON.

[bottleofwater]

@thomasjo That's actually what I do - I load my grammar via plist.parse, then create the grammar via:

let grammar = grammarRegistry.createGrammar(null, plist.parse(grammarString));
grammarRegistry.addGrammar(grammar);

Unfortunately, it then fails to properly tokenize my lines :(

[bottleofwater]

I've been able to track down the issue to this regexp (and others similar):

(?x)
  !(?!=)| # logical-not     right-to-left   right
  \|\| |  # logical-or      left-to-right   both

As you can see, these regexp have a trailing pipe that make them match everything. I'm not sure what to do - on one hand, it's obviously a coding mistake to leave this trailing pipe, but on the other hand it seems to work fine with both SublimeText and TextMate (I guess they discard every matching rule with 0-tokens as "not matching"), on a quite popular grammar. What do you think?

[bottleofwater]

Related issue on VSCode side: microsoft/vscode-textmate#9

[bottleofwater]

The JavaScriptNext syntax apparently uses the official Sublime's PackageDev in order to generate its tmLanguage files from YAML grammars, and it seems that this tool is the one adding trailing pipes. Fixing this would require updating every tmLanguage file generated by this tool up to this point. It isn't really feasible, so I feel like that it should be fixed on the first-mate codebase instead. My two cents.

My bad, the issue is actually in the YAML files themselves.

[rattrayalex]

@bottleofwater I ran into the exact same issue. Removing the trailing | for logical-or, assignment, and assignment.augmented helped, but didn't completely fix it. Were you able to find a full solution?

[rattrayalex]

Answering my own question, this line seems to have been the other problem, though I'm still not sure why. It contains this regex:

(?<!\.)\b(exports|module(?:(\.)(children|exports|filename|id|loaded|parent)))?\b

(note that I'm working with a fork of the repo @bottleofwater is using)

[winstliu]

It's effectively matching \b\b and checking that there is no period before it, which would be a zero-length match.
Maybe they meant (?<!\.)\b(exports|module(?:(\.)(children|exports|filename|id|loaded|parent))?)\b instead?

[arcanis]

I've also been faced with this issue, so I've opened a PR here: babel/babel-sublime#303

I'll backport your fix there, but I encourage you to manifest your interest for merging this feature in the upstream repository - I haven't had any feedback yet :(