Skip to content

fix: return EOF token instead of raising IndexError in TokenList.get_token#37

Open
yogendra-17 wants to merge 1 commit intoarxlang:mainfrom
yogendra-17:feature-checkLen
Open

fix: return EOF token instead of raising IndexError in TokenList.get_token#37
yogendra-17 wants to merge 1 commit intoarxlang:mainfrom
yogendra-17:feature-checkLen

Conversation

@yogendra-17
Copy link
Contributor

Pull Request description

TokenList.get_token() directly indexes self.tokens[self.position] without a bounds check. Calling get_next_token() past the end of the token list raises an unhandled IndexError with no source location, making it hard to debug and leaving the parser with no clean recovery path.

How to test these changes

  • ...

Pull Request checklists

This PR is a:

  • bug-fix
  • new feature
  • maintenance

About this PR:

  • it includes tests.
  • the tests are executed on CI.
  • the tests generate log file(s) (path).
  • pre-commit hooks were executed locally.
  • this PR requires a project documentation update.

Author's checklist:

  • I have reviewed the changes and it contains no misspelling.
  • The code is well commented, especially in the parts that contain more
    complexity.
  • New and old tests passed locally.

Additional information

Reviewer's checklist

Copy and paste this template for your review's note:

## Reviewer's Checklist

- [ ] I managed to reproduce the problem locally from the `main` branch
- [ ] I managed to test the new changes locally
- [ ] I confirm that the issues mentioned were fixed/resolved .

Copy link
Contributor

@xmnlab xmnlab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yogendra-17 could you write a test for that?
in the tests it shows already the token EOF, so I am not sure if it is the way to go ... maybe there is another thing that should be fixed instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants