Runtime: stream serialization of state at the end of each step

[josephjclark]

I think we've seen a case where a huge dataclip has triggered an OOMKill by kubernetes

Now, a Run executes in a child process with a limit of usually 1gb, or whatever. The run should be oomkilled if that child process exceeds its memory limit, and nothing is lost.

But that OOMkill does require memory allocation to run. Blocking processes, like JSON.stringify, could prevent the nodeprocess from blowing itself up.

So what can happen is this:

• A step completes and writes a huge dataclip to state. 1gb of JSON, why not
• The runtime serializes the state at the end of each step.
• For a large object, this serialization could use a lot of memory (i think json needs 3-4x the memory of the object its parsing)
• And if this serialisation is blocking, it'll just chew up available memory without blowing up
• And if the pod uses too much memory, it'll be instantly and gracelessly killed by kubernetes

So I think I need to look closely at that serialisation code. I think I use fast-safe-stringify right now - but if it's blocking, maybe I need to borrow the streaming serializer from the worker

Something else we might consider is saying: if the state object exceeds a certain limit in size, don't bother serializing, just pass it through. In a similar way that I want an exemption for streams on state.

[josephjclark]

At first glance this is looking super tricky.

If I use the steaming json parser used by the engine, we get different behaviour in terms of our state. That's because json-stream-stringify and fast-safe-stringify have different means of handling certain structures.

For example: circular references. Right now if the runtime hits a circular structure, it'll just chop it out with a string constant [circular]. Fine. If I use the streaming function, it'll handle the circle differently, or throw.

There are likely to be other subtle behavioural differences.

How much do we care about this? I don't care about the circular string. But I don't want to throw on circular structure detected, nor do I want to serialize circular data.

[josephjclark]

All really want is to run fast-safe-stringify aynchronously. Let the node runtime throw an OOM if it exceeds its memory limit. But that doesn't seem possible.

[josephjclark]

Ok, I may have figured out how to handle those tricky cases. We might be alright...

[josephjclark]

Ok, no - we can't use json-stream-stringify for this. There's a bug in cycle detection which will break real workflows.

So, what's next?

[josephjclark]

Options I can see:

1. Use the ensure payload size function verbatim, and just throw for any state object over a limit.
2. Don't bother with json.stringify - just rebuild the object removing all keys we don't want. This is easy for known things like functions and promises. What about circular references? How do we detect those?

Claude has little helpful to offer. It recommends using json streaming.

[josephjclark]

Discussed in depth with @taylordowns2000 - I think the best solution is to use ensurePayloadSize to guesstimate whether the state object is too big, and throw a controlled error to tell users that their state object is too large

Data clip limit is a % of run memory (maybe 25%?) or a fixed size. Or maybe the value is like 25% vs 200mb