Backtracking doesn't work repeatedly

[justheretostealcode]

Hi,
I get an error when I try to use the itergen project for generating python code. Is this possibly related to structuredllm/syncode#212 ?
This is the code to reproduce the problem:

from itergen.main import IterGen

# Define the grammar
grammar = "python"

# Initialize IterGen with the grammar and a model with Hugging Face model ID
iter_gen = IterGen(grammar=grammar, model_id="microsoft/phi-2", recurrence_penalty=0.1, max_tokens=200, parse_output_only=True)
prompt = "write me a fibonacci function in python with dynamic programming concepts. Just print the python code and no additional explanation or markdown."

# Start generation
iter_gen.start(prompt)


# Generate one sentence
generated_sentence = iter_gen.forward()
print("Generated Sentence:", generated_sentence)

# Backtrack by 2 words
current_words = iter_gen.backward("token", num=120)

# Inspect all words in the current generation
print("Current Words:", current_words)

Error:

IndexError                                Traceback (most recent call last)
Cell In[1], [line 15](vscode-notebook-cell:?execution_count=1&line=15)
     11 iter_gen.start(prompt)
     14 # Generate one sentence
---> [15](vscode-notebook-cell:?execution_count=1&line=15) generated_sentence = iter_gen.forward()
     16 print("Generated Sentence:", generated_sentence)
     18 # Backtrack by 2 words

File ~/Desktop/coding/SDT/itergen_last/itergen/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py:120, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    117 @functools.wraps(func)
    118 def decorate_context(*args, **kwargs):
    119     with ctx_factory():
--> [120](https://file+.vscode-resource.vscode-cdn.net/home/alex/Desktop/coding/SDT/itergen_last/itergen/~/Desktop/coding/SDT/itergen_last/itergen/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py:120)         return func(*args, **kwargs)

File ~/Desktop/coding/SDT/itergen_last/itergen/itergen/main.py:201, in IterGen.forward(self, unit, units, num, **gen_args)
    198 gen_mode = self._get_generation_mode(self.generation_config)
    200 # Initialize the parse results
--> [201](https://file+.vscode-resource.vscode-cdn.net/home/alex/Desktop/coding/SDT/itergen_last/itergen/~/Desktop/coding/SDT/itergen_last/itergen/itergen/main.py:201) parse_results = [ip.get_acceptable_next_terminals(self.structured_gen[idx]) for idx, ip in enumerate(self.inc_parsers)]
    202 initial_char_counts = [len(self.structured_gen[idx]) for idx in range(self.num_outputs)]
    204 unfinished_sequences = torch.ones(self.num_outputs, dtype=torch.long, device=self.device)

File ~/Desktop/coding/SDT/itergen_last/itergen/itergen/syncode/syncode/parsers/python_parser.py:95, in PythonIncrementalParser.get_acceptable_next_terminals(self, code)
     91         remainder_state = RemainderState.COMPLETE
     92 else:
...
---> [95](https://file+.vscode-resource.vscode-cdn.net/home/alex/Desktop/coding/SDT/itergen_last/itergen/~/Desktop/coding/SDT/itergen_last/itergen/itergen/syncode/syncode/parsers/python_parser.py:95)     current_term_str = self.parsed_lexer_tokens[-1].value
     96     remainder_state = RemainderState.MAYBE_COMPLETE
     97     final_terminal = self.parsed_lexer_tokens[-1].type

IndexError: list index out of range