Skip to content

Conversation

@shubhamugare
Copy link
Collaborator

@shubhamugare shubhamugare commented Dec 24, 2024

This PR includes two main changes:

  1. There are some fixes in parsers when the generation starts with empty output in the case of NL to code tasks for general programming languages.

  2. The parsing error and switch to unconstrained generation on an error is made more explicit. -- On any parsing error SynCode typically falls back to unconstrained generation (unless dev_mode flag is on). In the case of the grammar_mask mode, a parsing error can occur when the SynCode generation diverges from the grammar as SynCode does not always guarantee adherence to grammar.

Test:

python3 syncode/infer.py --mode grammar_mask --model meta-llama/Llama-2-7b-hf  --dataset humaneval --grammar python --max_new_tokens 400 --parser lr --parse_output_only False

results in pass@1 of ~12%

@shubhamugare shubhamugare merged commit 12a5e28 into main Dec 25, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants