Compliance validation

[arichiardi]

I am looking at this project and missionary with great interest on the ClojureScript lumo side.

I was wondering whether you plan to add specs to this project and/or some higher order abstraction that it runtime independent.

Great initiative, will keep following it.

[leonoel]

I don't have any plans yet, but that's definitely a path worth exploring. Feel free to share your thoughts about it.

[arichiardi]

Well you have a couple of them already, so for starters I guess a couple of s/fdef would make clear what a task is for instance. A little bit like https://github.com/cognitect-labs/anomalies I guess.

[leonoel]

I like this idea but I'm still unsure what would be the best approach. Here's what I got so far :

(ns task.spec
  (:require [clojure.spec.alpha :as s]
            [clojure.spec.gen.alpha :as g]))

(def canceller (s/fspec :args (s/cat) :ret any?))

(defn continuation [result-spec]
  (s/fspec :args (s/cat :result result-spec) :ret any?))

(defn task [success-spec]
  (s/fspec :args (s/cat :success (continuation success-spec)
                        :failure (continuation any?))
           :ret canceller
           :gen #(g/fmap (fn [x] (fn [! _] (! x) (fn []))) (s/gen success-spec))))

Random thoughts :

• clojure.spec is not sufficient to fully cover the literate specification. Some behaviors can only be validated by specific instrumentation (e.g ensuring continuations are not called more than once), and some others are hard to test due to nondeterminism (e.g behavior of cancellation)
• the default generator will only produce a restricted set of possible tasks : those who succeed synchronously. I wonder if it would be valuable to have a way to generate tasks completing asynchronously, or tasks randomly failing

[arichiardi]

Yes you are right, the specs would work if each state transition in the system can be represented as data so you end up specifying the "state" as a vector of maps. Is the underlying implementation logging state somewhere for replayability/ idempotency?

I should have looked at it first, before opening this issue probably.

[leonoel]

The task specification doesn't require implementations to expose state as values. The only piece of state that could be common to all task implementations is the completion status, say :pending, :pending-cancelled, :completed-success, :completed-failure. Maybe the specification could require the canceller to support deref to allow status polling. This path leads to reinventing CancelableFuture and this doesn't really help us to check correctness of the implementation, because the state machine can be arbitrarily more complex and receive events from arbitrary sources (for instance, in the clj-http wrapper example, the network event completing the tcp stream can happen concurrently with a cancel event, and the underlying state machine transitions are completely encapsulated under several layers so the only way to know whether request terminated normally or because of a cancellation is to check the final result).

I don't really see how a generic validation strategy could be possible in purely functional style. The task specification is inherently imperative because it is callback-based. This is on purpose, callbacks are required for any asynchronous communication and the point is to normalize these low-level technical artifacts in order to have a solid ground on top of which higher-level functional building is possible.

In my opinion, the good way to tackle this problem is to provide a TCK that could be used by implementers as a compliance test suite. That's the approach taken by RS, which is also a literate specification based on callbacks. I have renamed this issue, clojure.spec is interesting but it's a means, not an end.

[arichiardi]

I definitely now see why clojure.spec would not solve there. I am myself still looking for a way to effectively test async computations that might have errors or timeout is either step.

At the moment my implementations are protocol based where my "skeleton"/algorithm steps are all async functions - returning promises - that are replaced at test time with either failures or setTimeout longer than my maximum.

Every step returns the same kind of error so that I can homogeneously branch.

This feels still a bit itchy 😄

I am yet to find a better method of testing async than marble testing in RxJS, which, for me, was kind of effective.

Thanks for answering in such a detailed way. I am going to research more on the topic, not being an expert, and try to write back here possible ideas.