decodeTokens is missing its 'scopes' arguments in the doc

[arcanis]

It seems the last release added an extra parameter to pass to decodeTokens - without it, the decode process might fail with an error similar to this one:

Error: Expected popped scope to be meta.function.method.js, but it was undefined
    at GrammarRegistry.module.exports.GrammarRegistry.decodeTokens (/home/arcanis/cid/node_modules/first-mate/lib/grammar-registry.js:232:19)
    at Object.<anonymous> (/home/arcanis/cid/x.js:15:14)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:420:7)
    at startup (bootstrap_node.js:139:9)

Btw, why is that? Is it to avoid the need to store the scopes' enter tags at the beginning of each tokenized line? Because unless I'm mistaken, it also means that one has to iterate over every previous line in order to know what are the scopes for any random line.

[winstliu]

Looking at the latest changes, it doesn't look like anything changed with regard to decodeTokens. Can you give the first version where the exception occurs?

[arcanis]

Yep I was looking at the blame and it seems to have been that way for quite a long time. Weird, I thought it wasn't required before .. :|

That being said, my point about the perf hit still stands, I think. Have I missed something? Is there a builtin way to decode tokens that doesn't require to decode every previous line? I looked at the Atom source code, and apparently the decode process is done manually, which might be why you've never experienced this issue before.

[arcanis]

Oooooh I think I got it. There's an extra undocumented field called openScopeTags inside the structure returned by first-mate.