Adaptation of LM_scoring.py

[anthdr]

The LM_scoring.py script returns an error when called:

Traceback (most recent call last):
  File "/nas-labs/MT/pytorchwork/Recipes/toy-antoine/eole/eole/bin/main.py", line 38, in <module>
    main()
  File "/nas-labs/MT/pytorchwork/Recipes/toy-antoine/eole/eole/bin/main.py", line 34, in main
    bin_cls.run(args)
  File "/nas-labs/MT/pytorchwork/Recipes/toy-antoine/eole/eole/bin/tools/LM_scoring.py", line 68, in run
    config = PredictConfig(**config)
  File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 3 validation errors for PredictConfig
bin
  Extra inputs are not permitted [type=extra_forbidden, input_value='tools', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/extra_forbidden
sub_bin
  Extra inputs are not permitted [type=extra_forbidden, input_value='LM_scoring', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/extra_forbidden
config
  Extra inputs are not permitted [type=extra_forbidden, input_value='inference.yaml', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/extra_forbidden

(config.update(stuff_to_update) adds {'bin': 'tools', 'sub_bin': 'LM_scoring', 'config': 'inference.yaml'} which are unnecessary args for scoring)

I adapted the loading of the model, the vocab new logic and the computation of the loss as well.
To test it, I'm using a LM built with the wiki-103 recipe, I only changed training data (https://github.com/eole-nlp/eole/blob/main/recipes/wiki_103/wiki_103.yaml)

[vince62s]

you need to install last versions of black, flake8, pyproject-flake8 like here:
https://github.com/eole-nlp/eole/blob/main/.github/workflows/push.yml#L28-L35

[anthdr]

I did not call the script correctly, there is an error when called properly (modified initial comment accordingly).
While this script needed adaptation (pad_token retrieval, lambda opts are now elsewhere in the config, loss function now returns estims), I had to also modify loss computation mainly because one expected dimension is not there anymore, which made me realize I maybe overgeneralized loss/ppl computation to the specific case of wiki103 type of model.
Should loss computation use reduction="sum" to handle tensors of different dimensions ?

eole/eole/utils/loss.py

Line 107 in d2992b7

reduction="sum",

Moreover, I do not replicate the valid perplexity observed in my training and I don't know why yet:
tensorboard: 20.9643 (3.0428 loss)
lm_scoring: 22.34 (3.11 loss)