Skip to content

Conversation

@shubhamugare
Copy link
Collaborator

@shubhamugare shubhamugare commented Jan 5, 2025

The accept sequences with ignore terminals were resulting in retaining grammatically incorrect tokens in some cases.

Test:

python3 syncode/infer.py --mode grammar_mask --model meta-llama/Llama-2-7b-hf  --dataset humaneval --grammar python --max_new_tokens 400 --parser lr --parse_output_only False

results in pass@1 of ~13.41%

python3 syncode/infer.py --mode grammar_strict --model google/gemma-2-2b-it --dataset json_eval --grammar json --max_new_tokens 400 --parser lalr --device cuda:1

results in 98% accuracy

Fixes #143

@shubhamugare shubhamugare merged commit 1212098 into main Jan 6, 2025
1 check passed
@shubhamugare shubhamugare deleted the fix11 branch March 6, 2025 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot mask Name after Name in Python

2 participants