Onyx patch versions should not require new tenancy-id

[sundbry]

Upgrading Onyx versions between patches shouldn't require the user to use a new tenancy id. If the cluster-log format changes in an incompatible way, that should require a minor or major version number change. Otherwise, it can be very troublesome migrate job checkpoints, state, etc. to a new tenancy-id.

clojure.lang.ExceptionInfo: Incompatible versions of the Onyx cluster coordination log.
 A new, distinct, :onyx/tenancy-id should be supplied when upgrading or downgrading Onyx. 
{:cluster-version "0.13.3-alpha4", :peer-version "0.13.5"}

I propose that we go ahead and modify the check to ignore differences in major.minor.PATCH versions.

[lbradstreet]

I think this is a good idea, and I’ve considered it in the past. Ideally we could have a way to review a diff for the log files as part of the release script to prevent breaking log changes being included in minor releases.

…

On 27 Sep 2018, at 11:33, Ryan R Sundberg ***@***.***> wrote: Upgrading Onyx versions between patches shouldn't require the user to use a new tenancy id. If the cluster-log format changes in an incompatible way, that should require a minor or major version number change. Otherwise, it can be very troublesome migrate job checkpoints, state, etc. to a new tenancy-id. clojure.lang.ExceptionInfo: Incompatible versions of the Onyx cluster coordination log. A new, distinct, :onyx/tenancy-id should be supplied when upgrading or downgrading Onyx. {:cluster-version "0.13.3-alpha4", :peer-version "0.13.5"} I propose that we go ahead and modify the check to ignore differences in major.minor.PATCH versions. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[sundbry]

What do you think about decoupling the internal data version from the library API version?
In other words, if we ever want to break the tenancy log data structure in an incompatible way, we should increment the internal data structure version. However, this should probably be decoupled from the actual project version, since for example, an upgrade to 0.14.0 or even 1.0.0 might be perfectly compatible with the existing data structures even if there are incompatible API changes or new features.

We might be able to accomplish this in the present form, without too much code, by freezing the version in onyx.peer.log-version and only changing that when the internal data structures change moving forward.

As you mentioned about checking for breaking changes, a good point to start may be to identify which of the schema definitions are used in the persisted data structures. We might be able to programmatically detect then when those schema definitions change. Something like

(def peer-log-schemas {....})

(diff peer-log-schemas previous-peer-log-schemas)

Ultimately though, it will still just be a matter of good design discipline and code review to make sure the internal data versions stay compatible.