-
-
Notifications
You must be signed in to change notification settings - Fork 132
Description
We currently save model outputs (SS_vars.pkl, TPI_vars.pkl, and model_params.pkl) as pickle files. This issue is to discussing moving away from this file format -- likely towards a JSON format.
The disadvantages of pickle include:
- Data need to be loaded into Python.
- They are binary files and thus have an opaqueness that makes it difficult to view changes to elements in the data and could pose a security risk.
- Pickle files, particularly those saved with
cloudpickle, scubasmodel_params.pkl, are not guaranteed to open on versions of Python they were not saved in. This can inhibit the sharing of files and the long-term storage of data.
Using a JSON format would solve these three issues. SS_vars.pkl and TPI_vars.pkl contain only dictionaries of scalars, arrays, and strings, and so are easily converted to JSON.
Model parameters can almost always be saved to JSON (e.g., as in done with ogcore.utils.param_dump_jsonfunction). The exception to this is iftax_func_type=mono(ormono2D`), in which case the tax functions are SciPy interpolation objects. I am not aware of a way to put these into JSON. Perhaps JSON can be used in general, but a pkl can be used if tax functions of these types are used?