Process for 'catching up'

[cliftonc]

Because of the primarily event based architecture, we have a challenge for new systems added to the ecosystem, that want to synchronise all the data they need.

Best example perhaps is a new LA that wants all the SIS Data after setup.

Options:

1. Have the event log hold all events from all time (including every historical create event) - over time this will get very big.
2. Create a 'dummy' event log that holds create events for the current state of all entities
3. Create 'list' endpoints to allow retrieval of all entities
4. Create a 're-publish' endpoint that requests the service to create an event in the event log for all current state of entities, though all subscribers then receive and re-process it.

@EvanderVeen @niesink @mcginkel @dpronk thoughts?

[mcginkel]

As we have setup flows we can also link it to those:
example: S.2.0 Administrator establishes relationship between SIS, LMS and Marketplace.
When a MP admin asks access to the SIS, the school administrator of the SIS for that school approves. At that moment the SIS can send all events needed to get in sync with the current state to that MP. It will be a lot of events, but we will use the same mechanism for updating information.

The SIS can also define at that point what relevant data is. For example it will only send the actual data for this year, not for the previous years.
It also prevents security issues that you need to secure endpoints to only allow data that is relevant. We can limit those endpoints now to all data that the SIS has send an event before.