BaselineCLS on Delta

[dougbrn]

Using this issue to summarize some outstanding issues on running HyraxBaselineCLS on delta.

The first point to make is that the core training/inference loop has worked on delta, but the implementation that performs pre-training has not been fully validated. Some issues seemed more general to Hyrax, and potentially it's integration with my setup which was running Python 3.9:

1. I was getting fairly low level dtype errors that were spun off of hyrax set_config calls, if encountering these you may be able to get around them by instead updating configs using the following assignment syntax h.config["path"]["to"]["key"] = new_value
2. The pre-training implementation is setup as a separate hyrax instance that is run after the main instance is setup but before the main instance is trained, this was potentially affecting some metric emission behavior, if encountering that I would recommend splitting the script into a dedicated pre-trainer script and a dedicated train-with-weights script