Additions to the EKP struct should be considered breaking changes

[Julians42]

I am excited by the new functionality that #482 offers - it should streamline simulation evaluation. However, by adding to the EKP structure here, it is "breaking" in the sense that I cannot read any EKP from an experiment run with an older version of EKP, even v2.4.0. As I've run several simulations with v2.4.0 that are rather expensive evaluation will be harder. I was wondering if:

1. We actually need to store em in the EKP object as opposed to just returning a dict when called.
2. We can bump minor versions as opposed to patches on breaking changes. Does CliMA have a policy for where breaking changes should be introduced? @charleskawczynski
3. We can fix this general problem where we can't read previously run experiments either with additional functionality (e.g. something like a read_backwards_version function) or changing how the EKP object is loaded. Currently JLD2 is not robust to these changes and perhaps we could load this with a failsafe method. (Or perhaps this exists already and I've missed it)

[ph-kev]

For the second question, I believe that we just follow semantic versioning (see here), so the minor version number should be updated if it was a breaking change. Although, I wouldn't call the change breaking for EnsembleKalmanProcesses.jl and maybe a breaking change for ClimaCalibrate.jl depending on whether ClimaCalibrate promises backward compatibility with previous versions of the ekp struct.

I am also not sure if this issue belongs here or in ClimaCalibrate.jl, since EnsembleKalmanProcesses.jl doesn't use JLD2. I think it make sense to make a function like read_backwards_version in ClimaCalibrate.jl, but I don't know how feasible and robust that would be.

[odunbar]

I'm sorry that you have encountered this @Julians42 , JLD2 is a real pain when it comes to versioning - as we have found out repeatedly...

As far as I'm aware from the definition of "breaking changes" is when the API has changed and the user has to change their code for it to keep running. An actual breaking change would if, say, we had changed get_error(ekp) to return the new dict, and not a vector any more. This would cause possible breaks where users code expects a vector but recieves a dict - for this reason instead we added a new getter get_error_metrics(ekp) returning a dict so that get_error can remain compatible with a vector. In this sene we don't have it breaking in that sense, as you can run the same scripts with 2.4.0-2 without error.

Unfortunately the maintenance required to manage a read_backwards_version is very heavy, I don't think is practically maintainable.

As this is likely to happen a lot in the future too (where internals are changed and it causes HDF5 issue) I wonder if we should not think specifically about error_metrics but think of more general solutions:

One idea:

We encourage the users to save all the primary data objects (u,g,observationseries,...,\Delta t, priors) separately.

These unfortunately then will not give you access to the nice plotting/linear algebra functionality of EKP, but may allow you to access data more easily (NB there may still be issues with e.g. priors or observation series)

Second idea:

rather we could add three new functions

1. dump_internals_to_jld2(ekp, filename) # saves the fields of ekp separately
2. dict = load_internals_from_jld2(ekp, filename) # loads the fields into a dict
3. ekp = rebuild_ekp_from_dict(dict) # builds the ekp object from a dict

This makes it less likley to face a load error from jld2; instead one will face an error when building from fields in 3. This takes in a dict, so the user can hack the dict to make it compatible until new runs are completed. In theory this is still fallible when loading from 2. but likely will happen far less often as it is another tier down the serialization tree.

Third idea:

We dump everything in HDF5, completely serialized (but this will not preserve information about the structs etc.) One can also write (complex) custom deserializations in JLD2 but unsure how easy this would be fore large heterogenous structs

Other ideas to follow...