Skip to content

Moving away from pickle files for model output #1025

@jdebacker

Description

@jdebacker

We currently save model outputs (SS_vars.pkl, TPI_vars.pkl, and model_params.pkl) as pickle files. This issue is to discussing moving away from this file format -- likely towards a JSON format.

The disadvantages of pickle include:

  1. Data need to be loaded into Python.
  2. They are binary files and thus have an opaqueness that makes it difficult to view changes to elements in the data and could pose a security risk.
  3. Pickle files, particularly those saved with cloudpickle, scubas model_params.pkl, are not guaranteed to open on versions of Python they were not saved in. This can inhibit the sharing of files and the long-term storage of data.

Using a JSON format would solve these three issues. SS_vars.pkl and TPI_vars.pkl contain only dictionaries of scalars, arrays, and strings, and so are easily converted to JSON.

Model parameters can almost always be saved to JSON (e.g., as in done with ogcore.utils.param_dump_jsonfunction). The exception to this is iftax_func_type=mono(ormono2D`), in which case the tax functions are SciPy interpolation objects. I am not aware of a way to put these into JSON. Perhaps JSON can be used in general, but a pkl can be used if tax functions of these types are used?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions