Epic 5: Change post-processing to Python + numpy code

[jonc125]

• Rather than using our post-processing DSL, allow protocol authors to use plain Python code plus numpy operations on the n-d arrays, thereby avoiding all interpretative overhead and allowing more flexible analysis, control flow, error handling, etc.
• Initially we'll only have trusted users; later we'll need to consider further security
  • Restricting what code is allowed; if just core python (no stdlib) plus numpy operations we might be OK
  • Isolate execution runs using e.g. Docker
• Possibly consider an editor GUI for protocols - at least syntax highlighting - if easy to do!

[MichaelClerx]

Maybe we can re-open discussion about this? Not sure if it'd be the best use of our time if we just want to get the job done (without too many changes to how it works).
Things like sandboxing and creating a GUI will be quite time-consuming I think. And people get very picky about plots, so probably best done offline, in an environment of the user's choice?

A good candidate for leaving till WL3 I think

Thoughts @jonc125 @mirams ?

[mirams]

I think spending some time making the most un-usable bit of the WL more usable is a good idea, and that is definitely multi-dimensional array operations on the N-Dim output arrays i.e. just the postprocessing block of the protocol.

If the 'fitting spec' part is going to be sandboxed python/Pints isn't this 'post-processing' being sandboxed Python/numpy a pilot for that?

A GUI for the full protocol edit is really a separate thing to the main thrust of this Epic?

[MichaelClerx]

I think spending some time making the most un-usable bit of the WL more usable is a good idea, and that is definitely multi-dimensional array operations on the N-Dim output arrays i.e. just the postprocessing block of the protocol.

That structure is going to be the same no matter what language you tackle it in. The FC syntax to work with arrays is basically the Python syntax, no?

[MichaelClerx]

If the 'fitting spec' part is going to be sandboxed python/Pints isn't this 'post-processing' being sandboxed Python/numpy a pilot for that?

Is it? I thought the spec would just tell us what Python/PINTS code to generate?

[jonc125]

That structure is going to be the same no matter what language you tackle it in. The FC syntax to work with arrays is basically the Python syntax, no?

No, the FC syntax is much harder to work with, as arrays are immutable and you need to think in functional programming terms. Your typical researcher is much more familiar with numpy operations. Running numpy directly would also be more efficient in general (as you're not interpreting the FC syntax).

Certainly extensions like sandboxing, editor, etc. are probably not going to be reached in this grant. But at the Oxford meeting we did discuss allowing either trusted users to run arbitrary Python, and/or a simple syntactic check on post-processing to make sure no potentially dangerous code was included.