Support annotations

[SimonWoolf]

Spec equivalent of ably/ably-js#1953
PUB-1549

[sacOO7]

Wow, big PR, will take some time to review it properly 👍
Thanks for adding !

[lawrence-forooghian]

Reviewed everything except for the docstrings; will look at those shortly.

[lawrence-forooghian]

In the JS typings it seems like this type does aim to be exhaustive — it doesn't offer the possibility of receiving an arbitrary JSON object in the Message.summary property.

[SimonWoolf]

This is kindof a limitation of typescript. I initially had type SummaryEntry = SummaryDistinctValues | ... | unknown. The trouble is that unknown is a collapsing type; x | y | unknown is just equivalent to unknown -- which means you don't get any type hints, autocompletion etc. So I ended up removing the | unknown, and expecting we'd have to tell people in the docs to use (msg.summary as any).xxx for summary types the ably-js types don't include yet.

If you happen to think of any better suggestions for that than what I did that I'm all ears, I realise it's not great.

[lawrence-forooghian]

which means you don't get any type hints, autocompletion etc

I'm not sure I understand to what extent you currently get these (i.e. even without the unknown)? Presumably, given a SummaryEntry, the only thing that autocompletion can offer you are properties which exist on the union of all possible SummaryEntry types, which doesn't seem hugely useful?

Perhaps the thing to do here in TypeScript is that which resembles option 2 of #292 (comment) — i.e. keep SummaryDistinctValues etc but remove SummaryEntry and change Message.summary to a JSON object?

[SimonWoolf]

Yeah, fair point.

You do get one thing with the current typings, which is you can use go-to-type to go to the relevant bit of the type definitions and look at them. Adding | unknown kills that: the type is collapsed into unknown and drops all metadata. But now I come to think more, a better solution would be | {[k: string]: unknown}. (Which is essentially JSONObject).

Not sure I see the argument for removing SummaryEntry as a union type, it gives you that thing and doesn't really harm anything. Either way you'd still get that option 2 through a type assertions.

[lawrence-forooghian]

Ah, so if you go with the {[k: string]: unknown} then it doesn't collapse it, I see. That sounds good. Might be worth adding a comment alongside the TypeScript type definition to explain why the union type exists; I can quite easily see someone otherwise coming along and going "this type adds no value, I'll remove it".

[SimonWoolf]

Adding more notes to the ably-js typings seems useful sure. (Also considering providing type assertoin functions). But unless I'm misunderstanding, you're not suggesting any change to this spec PR, so that can happen async in the ably-js repo.

[lawrence-forooghian]

I worry that we're pushing quite a lot of work onto users by using such a loose type here. It's not too bad in an untyped language or something like TypeScript where you can tell the compiler "assume that this JSONObject has this given (potentially nested) shape". But there isn't a similar mechanism in a language like Swift in which the type system cannot represent arbitrarily-shaped JSON objects; we'd need to give the user a dictionary with untyped values and they'd need to convert it into some strongly typed object they can work with (i.e. something that looks like one of the SummaryEntry values that you've given in the docstrings). I think that expecting them to do this for non-user-supplied data (i.e. data whose shape Ably controls) would be quite a faff for them.

The ideal would be for the SDK to provide these possible SummaryEntry values as part of its public API, so that one of the following can happen:

1. the SDK, using a rule for mapping from a Message.summary key to a SummaryEntry value type, creates the appropriate SummaryEntry value type by calling a constructor that accepts a JSONObject (or returns the JSONObject if there's no rule for a given key)
2. same thing as 1. but the mapping is performed by the user (this one might be better so that the type we emit in the message summary doesn't suddenly change when we add support to the SDK for some new aggregation method)

[SimonWoolf]

I did consider 1 but, like you point out, it kindof sucks because it means that when we do add support the type changes suddenly in what's supposed to be a non-breaking library change. And we really do want to be free to add new aggregations serverside without having to coordinate half a dozen SDK releases each time.

But 2 seems much more reasonable, that's not a bad idea..

[lawrence-forooghian]

If you're happy with 2, then I think that the thing to do is to add the TypeScript types (SummaryDistinctValues, SummaryUniqueValues, SummaryMultipleValues, SummaryClientIdList, SummaryTotal) into the spec, and have a spec point explaining that there should be a mechanism for a user to convert a Message.summary into a user-selected one of these (this mechanism might just be a type assertion e.g. in TypeScript, or a constructor that takes a JSONObject e.g. Swift). What do you think?

[SimonWoolf]

Done, wdyt?
Slight tweaks to naming (added the version to the constructor name to match the version being in the annotation type) that I'll need to do in ably-js too.

[lawrence-forooghian]

Looks good, please could you update the docstrings and IDL to match?

[sacOO7]

Adding to this, can we also have public methods that returns boolean like
isSummaryDistinctValues, isSummaryUniqueValues etc, so it's easier for user to cast/convert into appropriate type. Same pattern is followed in java gson library

https://github.com/google/gson/blob/dd2fe59c0d3390b2ad3dd365ed6938a5c15844cb/gson/src/main/java/com/google/gson/JsonElement.java#L119-L149

I guess this has to follow pattern where Summary is a base class and SummaryUniqueValues, SummaryMultipleValues, ... extends Summary

class Message {
    // other fields
    Summary summary;  // type can be SummaryUniqueValues, SummaryDistinctValues etc
}

// Summary.java
public abstract class Summary {
    private Map<string, JsonObject>? value; // TM2q

    public boolean isSummaryDistinctValues() {
        return this instanceof SummaryDistinctValues;
    }

    public boolean isSummaryUniqueValues() {
        return this instanceof SummaryUniqueValues;
    }
   
    public SummaryUniqueValues getAsSummaryUniqueValues() {
        return (SummaryUniqueValues) this;
    }
}

// SummaryUniqueValues.java
public class SummaryUniqueValues extends Summary {
    // Implementation specific to unique values summary
}

// SummaryDistinctValues.java
public class SummaryDistinctValues extends Summary {
    // Implementation specific to distinct values summary
}

I think we can establish this standard pattern across SDKs. Currently, the usage of static factory methods is not entirely clear and related implementation is different in ably-js.

[sacOO7]

One more related comment -> https://github.com/ably/specification/pull/292/files#r2137836289

[SimonWoolf]

I guess this has to follow pattern where Summary is a base class and SummaryUniqueValues, SummaryMultipleValues, ... extends Summary

The pattern Lawrence proposed, and I then adopted, for languages where it is not easy to work with plain objects, is a function SummaryDistinctV1() that takes a JsonObject (the value of some key in the summary dict), and returns a structure that is specific to a distinct v1 summary. The user is expected to know what aggregation type they're using and call the correct function. In this pattern, there is no subclassing.

I think you're imagining an api where the library detects what kind of summary entry it is and does the decoding itself automatically, which would then need subclassing. That is a possible api we considered, but we decided against it.

In ably-js working with plain objects is easy and idiomatic, so the sdk just returns objects, and the user can use type assertions to get the correct type. (We could potentially also provide type assertion functions).

[SimonWoolf]

@lawrence-forooghian any further comments?

[lawrence-forooghian]

Was away for a couple of weeks — will take a look now.

[lawrence-forooghian]

Nothing beyond the current unresolved conversations.

[sacOO7]

Suggested change

	| summary: `Dict<string, JsonObject>`? ||| TM2q | A summary of all the annotations that have been made to the message, whose keys are the `type` fields from any annotations that it includes. For summary types the SDK knows about, static factor methods on `Message` exist to parse the values into strongly-typed objects. |
	| summary: `Dict<string, JsonObject>`? ||| TM2q | A summary of all the annotations that have been made to the message, whose keys are the `type` fields from any annotations that it includes. For summary types the SDK knows about, static factory methods on `Message` exist to parse the values into strongly-typed objects. |

[sacOO7]

Or can be updated to
A collection summarizing all annotations attached to the message, where each entry is keyed by the type field from the corresponding annotation. For known summary types, the SDK provides static factory methods on the Message class to parse these values into strongly-typed objects.

[ttypic]

Can we define this more loosely in the spec? Not require static factory methods on Message, but suggest providing some utility methods, idiomatic to the SDK, to perform these casts to strongly-typed objects. I’m just not sure that Message is the right place to put those static methods.

[sacOO7]

Yeah, in case of ably-java, we added them as static methods of Summary.java (PR => Java annotations) or can we define them as public utility methods, wdyt

[SimonWoolf]

Can we define this more loosely in the spec? Not require static factory methods on Message, but suggest providing some utility methods, idiomatic to the SDK, to perform these casts to strongly-typed objects. I’m just not sure that Message is the right place to put those static methods.

That's pretty much exactly what the spec already says. I can modify TM7 to make clear that it doesn't have to be static methods on Message to make that explicit, sure. (They won't be in ably-js for example).

[sacOO7]

Added few typo comments.
Other than this, do we need to mention that annotations introduces mutable messages.
I could see this change as a part of java annotations PR
AppSpec.json#Mutable=true

[sacOO7]

requested few changes

[SimonWoolf]

Other than this, do we need to mention that annotations introduces mutable messages.

Eh, that's effectively a detail on how this functionality is tested; I've never been a big fan of having the features spec specify that sort of thing, I don't think it's necessary. The features spec doesn't lay out exactly what any of the other namespaces created by test-app-setup.json fixtures are or what needs them, why would it for this?

[maratal]

Should it contain somethig similar to RestAnnotations? @SimonWoolf