Bug: Attribute used before being set in FinetuningTokenGenerator

[BenjaminCarpenter480]

Bug:
AttributeError: 'FinetuningTokenGenerator' object has no attribute 'pad_id' occurs whilst attempting to create preprocessor for a image-caption dataset.

I believe this is from the Cerebras modelzoo code for this snippet finetuning_token_generator.py:74-79

        self.eos_token = (
            self.tokenizer.convert_ids_to_tokens(self.pad_id)
            if self.eos_id is None
            else self.tokenizer.convert_ids_to_tokens(self.eos_id)
        )
        self.pad_id = pad_id

self.pad_id is being used just before its assignment.

In my case moving the assignment just above this snippet fixed the problem ensuring it was set before use. The error causing line self.tokenizer.convert_ids_to_tokens(self.pad_id) was added slightly later to this section so might explain where the issue has arisen from.