Define overall architectural approach for exchanges

[cliftonc]

We need to agree high level principles and approaches for how data is exchanged.

It needs to be: Reliable, Scaleable, Developer Friendly, Secure & Future Proof.

Initial list of exchange patterns:

Type	Description
Notify / Pull	Notification via web-hook, followed by pull of data from source.
Publish / Subscribe	Publication of feed of changes on source, destination pulls data as needed.	Alternate/additive model to Notify / pull via event feed.
Push	Direct call of web service on destination from source.
Pull	Direct call of web service on source from destination.
Browser	Exchange via URL data in browser

[cliftonc]

• Define the output of the data and exchange work

[wvholland]

Perhaps its obvious for all off you and already taken into account but ... did we also consider resilience as one of the constaints. The whole construction should not collapse if something is not working. We will design it in such a way that we know that at some times things will fail, and still the system keeps working ...

[cliftonc]

The following was notes from a conversation between @mcginkel @EvanderVeen and @cliftonc over the previous 2hrs.

Principles / Areas

• Timing - "real time" vs nightly / batch.
• Async vs Sync - mix of model, with focus on async notification to avoid nightly batch - synchronous as a fallback.
• UWLR - data sync in bulk; student added to group during the day - can't use LAs - this is an anti-pattern we need to avoid.
• If data doesn't exist that you need then give meaningful messages, or go and get the data you need
• Current standards assume certain data is always present, and exactly what you get - so not flexible.
• Assume minimum data - all other is optional? How do we deal with different consumers needing different data sets?
• Profiles related to Oauth flow - e.g. request data access that is approved - via scopes with recognisable names [talk in security session] - scopes should be connected to API design, e.g. separation of student / student address.
• Define needs / triggers for data processing agreements?
• Personas - e.g. LA for one subject / project vs larger platform
• Existing standards - lets be pragmatic and assess with each exchange

Patterns for integration

• Push changes
  • No polling - except as fall back
  • If you miss events you can come back and get them
  • Should the events contain data - this requires further discussion on pros/cons / are there rules?
• Get data when needed
  • How do we deal with batch / bulk data efficiently?
• Rate limiting / fair use
  • Service providers can define terms independently
  • How do we model relationships for more complex objects (groups <> students)

Approach to identifiers:

• Each providing system has a responsibility of providing a unique identifier for each data entity
• In some scenarios - e.g. learners - there may be additional 'external' identifiers such as the ECK ID
• Event feed should always reference that identifier in any communication

Proposal for Exchanges:

• Three parts:
  • Webhook subscription for 'live' events
    • Need to provide a way for subscriber to confirm message came from right source (e.g. https://developer.github.com/webhooks/securing/)
  • API to get past events (like an event stream - paginated)
    • Need to define the format and structure of the event stream
    • What is an event? Business or data event? - more likely to be data events 'user updated'.
  • API to get specific objects referenced in events
    • Rest as default
    • Graphql is useful here as it allows you to 'reach into' the tree of relationships

Examples from other places to explore

• Stripe https://stripe.com/docs/api/events
• Github https://developer.github.com/v3/activity/events/
• Twilio https://www.twilio.com/docs/usage/monitor-events
• https://www.asyncapi.com/ may be interesting to document along with https://www.openapis.org/

[EdwinIddinkGroup]

Security and data transmission must meet the standards as laid down in Edu-K / Edustandaard. This includes the use of / taking into account:

1. • The Edukoppeling transaction standard is part of the reference architecture for education (ROSA). Within which the use of REST and SOAP is guaranteed.
2. • Information Security and Privacy ROSA Certification Scheme.