Sync lexer punctuations with ZetaSQL

[apstndb]

Some punctuations are not yet handled in memefish.Lexer.
It cause panic when lexing even if parsing is not needed.

I think it is safe to add all punctuations in ZetaSQL.

https://github.com/google/zetasql/blob/194cd32b5d766d60e3ca442651d792c7fe54ea74/zetasql/parser/flex_tokenizer.l#L513-L552

• (
• [
• {
• )
• ]
• }
• *
• ,
• =
• += KW_ADD_ASSIGN
  • It seems to be used in options_assignment_operator
• -= KW_SUB_ASSIGN
  • It seems to be used in options_assignment_operator
• !=
• <=
• <<
• =>
• -> KW_LAMBDA_ARROW
  • Support lambda expressions #153
• <
• >
• >=
• ||
• |
• ^
• &
• +
• -
• /
• ~
• ?
  • Positional parameter is available in BigQuery, but Cloud Spanner has not yet supported.

    Positional parameters are not supported

• !
  • It seems not to be used.
• %
  • GQL Implement Spanner Graph queries #99
• |> KW_PIPE
  • Pipe syntax
• @
• @@ KW_DOUBLE_AT
  • System variables in BigQuery
• .
• :
  • NEW constructors Support protobuf NEW operators #119
  • GQL Implement Spanner Graph queries #99
• \ BACKSLASH
  • Only for lenient macro expansion
• ;
• ${unquoted_identifier} MACRO_INVOCATION
  • internal names in ZetaSQL?
• ${decimal_digits} MACRO_ARGUMENT_REFERENCE
• $
  • It seems not used.

Except MACRO_INVOCATION and MACRO_ARGUMENT_REFERENCE would be better to be implemented.

This makes Lexer more robust even to new lexical structures that are being tested privately.

[apstndb]

• All punctuations are added in Sync lexer punctuations with ZetaSQL #182