Keep API in-line with upstream

[amaanq]

Similar to py-tree-sitter's updates, not done yet but putting it up for early bird reviewers:

Todo:

• Complete dsl/index definitions
• Lookahead Iterator
• Tests for it all

@verhovsky you might be interested

[verhovsky]

One function that's sorely missing is childrenForFieldName, it looks like there's actually no such function in api.h, it's a Rust thing? but there is at least ts_node_child_by_field_name which would be good too, you could get the command_name of a command node in Bash, but an iterator version to get all the argument nodes is what I wanted. Here's a list of all the C functions in api.h that never appear in node-tree-sitter before the changes in this PR:

ts_language_field_id_for_name(
ts_language_next_state(
ts_language_state_count(
ts_language_symbol_for_name(
ts_lookahead_iterator_current_symbol(
ts_lookahead_iterator_current_symbol_name(
ts_lookahead_iterator_delete(
ts_lookahead_iterator_language(
ts_lookahead_iterator_new(
ts_lookahead_iterator_next(
ts_lookahead_iterator_reset(
ts_lookahead_iterator_reset_state(
ts_node_child_by_field_name(
ts_node_descendant_count(
ts_node_eq(
ts_node_field_name_for_child(
ts_node_grammar_symbol(
ts_node_grammar_type(
ts_node_is_error(
ts_node_is_extra(
ts_node_is_null(
ts_node_language(
ts_node_next_parse_state(
ts_node_parse_state(
ts_parser_cancellation_flag(
ts_parser_included_ranges(
ts_parser_language(
ts_parser_parse_string(
ts_parser_parse_string_encoding(
ts_parser_reset(
ts_parser_set_cancellation_flag(
ts_parser_set_timeout_micros(
ts_parser_timeout_micros(
ts_query_capture_count(
ts_query_capture_quantifier_for_id(
ts_query_cursor_delete(
ts_query_cursor_did_exceed_match_limit(
ts_query_cursor_match_limit(
ts_query_cursor_remove_match(
ts_query_cursor_set_byte_range(
ts_query_cursor_set_match_limit(
ts_query_cursor_set_max_start_depth(
ts_query_disable_capture(
ts_query_disable_pattern(
ts_query_is_pattern_guaranteed_at_step(
ts_query_is_pattern_non_local(
ts_query_is_pattern_rooted(
ts_query_start_byte_for_pattern(
ts_query_string_count(
ts_set_allocator(
ts_tree_copy(
ts_tree_cursor_copy(
ts_tree_cursor_current_depth(
ts_tree_cursor_current_descendant_index(
ts_tree_cursor_goto_descendant(
ts_tree_cursor_goto_first_child_for_point(
ts_tree_cursor_goto_last_child(
ts_tree_cursor_goto_previous_sibling(
ts_tree_cursor_reset_to(
ts_tree_included_ranges(
ts_tree_root_node_with_offset(

maybe some of these are also missing functions that should have bindings.

I generated it like this:

cd /tmp
git clone https://github.com/tree-sitter/node-tree-sitter
rg -o --no-filename 'ts_\w*\(' node-tree-sitter/src | sort | uniq > used_in_node.txt
curl https://raw.githubusercontent.com/tree-sitter/tree-sitter/master/lib/include/tree_sitter/api.h | rg -o --no-filename 'ts_\w*\(' | sort | uniq > api.h
comm -23 api.h used_in_node.txt

Thanks for working on this.

[amaanq]

Btw, do we want to switch to napi still? I'd like to do that upstream and here if that proves to be very beneficial, I forgot what exactly the benefits are rn

[verhovsky]

The benefits are that you don't need to upload as many binary files with each node-tree-sitter release, there's less work to get it to support each new Node version and something about making the extension "context aware" might be easier. And it's a newer API. It would be good to do

[amaanq]

Agreed, it's just massively breaking as all grammars will need to regenerate their bindings to have the napi one instead of nan

[amaanq]

I'm nearing the finish line for this PR (pretty much just LookaheadIterator left) - but that being said, I have no idea why CI is failing, it does require rebuilding the bindings instead of using the latest release for obvious reasons...

[amaanq]

LookaheadIterator is done - the only functions that are available upstream but not used here are the following:

ts_language_next_state(
ts_language_state_count(
ts_language_symbol_for_name(
ts_lookahead_iterator_language(
ts_node_eq(
ts_node_is_null(
ts_node_language(
ts_parser_cancellation_flag(
ts_parser_language(
ts_parser_parse_string(
ts_parser_parse_string_encoding(
ts_parser_set_cancellation_flag(
ts_query_capture_count(
ts_query_capture_quantifier_for_id(
ts_query_cursor_delete(
ts_query_cursor_remove_match(
ts_query_string_count(
ts_set_allocator(
ts_tree_cursor_copy(

Some are just not needed, and those related to language need changes upstream to allow the generated tree-sitter-xxx binding to expose these, as that contains the "Language" type, and isn't created here. We could create an extended method to say, call language_next_state(JavaScript, ...) but that's much worse than literally calling it on the language. I'd like it like Javascript.next_state(...) instead.

Tests are done, I just have to update the dsl/index.js and then this is ready

[amaanq]

alright it's done, do try it out thoroughly if you can

[verhovsky]

I get this type error trying to use this code:

../third-party/node-tree-sitter/tree-sitter.d.ts:214:7 - error TS2416: Property 'next' in type 'LookaheadIterator' is not assignable to the same property in base type 'IterableIterator<number>'.
  Type '() => number' is not assignable to type '(...args: [] | [undefined]) => IteratorResult<number, any>'.
    Type 'number' is not assignable to type 'IteratorResult<number, any>'.

214       next(): number;
          ~~~~

If I change this to

Suggested change

	next(): number;
	next(): IteratorResult<number>;

then it complains that

../third-party/node-tree-sitter/tree-sitter.d.ts:208:18 - error TS2420: Class 'LookaheadIterator' incorrectly implements interface 'IterableIterator<number>'.
  Property '[Symbol.iterator]' is missing in type 'LookaheadIterator' but required in type 'IterableIterator<number>'.

208     export class LookaheadIterator implements IterableIterator<number> {
                     ~~~~~~~~~~~~~~~~~

  node_modules/typescript/lib/lib.es2015.iterable.d.ts:53:5
    53     [Symbol.iterator](): IterableIterator<T>;
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    '[Symbol.iterator]' is declared here.


Found 1 error in ../third-party/node-tree-sitter/tree-sitter.d.ts:208

[amaanq]

superseded by #190