How To Do Serialization?

[BarbourSmith]

This is a related question to this question about caching but I didn't want to hijack the conversation there.

I would like to be able to run operations in different workers in parallel, but to be able to pass the replicad geometry back and forth to the workers it needs to be serializable.

Do you have any thoughts on how to best go about doing that? Would using .step representations for geometry work well or is that super inefficient?

[BarbourSmith]

Doing a little bit of digging it seems like the BREP format is probably more appropriate for serialization than using STEP.

I'm curious about how it would work in practice (would it be painfully slow or lead to other issues).

[sgenoud]

Yes, you are right. But in any case I would be a bit afraid of this - tell me what you find out!

[BarbourSmith]

I will give it a go and report back 😁

[tristan-huber]

Hi @sgenoud! I'm working with Bar on this serialization project and want to check in with you now that we've gotten a little further along.

So far BREP seems promising as an accuracy-preserving way to serialize 3d shapes and ship them between workers as needed. In our current proof of concept I'm using oc's BinTools.read and BinTools.write methods to serialize shapes directly to/from memfs files. Then replicad's Shape.cast to wrap the deserialized result. All that being said, it's still early days I'm not yet sure how fast this serialization/deserialization will be.

There's two main questions I have for you re replicad and this serialization functionality.

1. While brep looks good for 3d shapes, 2d drawings are represented entirely differently both in replicad and in oc. Their OC representations aren't a subclass of TopoDS_Shape which precludes using oc's brep serializer, and there's significant structure stored in the replicad layer (ie: the hierarchy of blueprints and curves). Do you have a sense of what serialization format would work well for Drawings?
   I don't have a strong sense between svg, step, etc. I could also imagine some version of this where we push 2d shapes into 3d and reuse the brep serialization, though this does seem complex and I'm not sure how we'd recover the 2d from the 3d.

2. Do you have any interest in serialization support being added to replicad itself?
   At a minimum I'd love to include BinTools in replicad so that we no longer need to use a forked version and other replicad users have access to the brep read/write functions. Or, if you're interested, I could work on implementing serialization within replicad, eg: as a method on Shape (3d) and on Drawing (2d). If you're interested in this or something similar I'm happy to write up a more detailed proposal.

[sgenoud]

I would be happy to review and merge some changes for serizalization in general!

About your 2D case, this will be a bit more difficult my guess is that you could make a json format that would support it. You might want to look at the type of the curve and extract its structure there.

Have a look at the way I export as SVG. But instead of using the SVG compatible approximation, extract the info needed to recreate the curve.

[tristan-huber]

Ok! proposal doc is here: https://docs.google.com/document/d/1OJe7Q-B2cLUjmPFJDHQe51yAeXRFBDN8w32epx0eI8s/edit?usp=sharing, would welcome comments especially on the public api (currently planning to add instance & static methods onto Drawing and Shape in replicad).

Good news on the 2D serialization front, I found this library in OC which seems to support read/write of Geom2D_Curve's. https://dev.opencascade.org/doc/refman/html/class_geom_tools.html

Biggest technical hurdle will be dealing with the Standard_OStream and Standard_IStream types which these OC library method's require. But, I think I've got a couple ideas about how to manage those. Will link the PR here once it's ready.

[sgenoud]

Thanks a lot! This looks great. Some comments:

• While I like the serialize as a method, I have not really been using static methods in replicad, I just use simple function there. I don't have anything per se against static method, but it would feel more consistant to have a deserializeShape and a deserializeDrawing method.
• I would not use items for both blueprints and blueprint object - calling it curves in the case of the blueprint makes it easier to read (that said, it does not change much).
• I don't think the choice of option changes much - in both cases you introduce some additional cpp code. I would go for the simplest options (probably option 2).

[tristan-huber]

Great! all these suggestions sgtm! Option 2 for the additionalCppCode did turn out to be substantially easier.

PR is now available! #216

mentioned there but I ran into one hickup around duplicating code between the yaml configs. Comments & feedback welcomed of course :)

[Billiam]

I was testing this in the hopes that serialized shapes could be used as cache keys, but noticed that there can be subtle differences between the original shapes, and the deserialized ones that appear to affect future cache operations.

Here's a comparison of a fuse operation's original output and a second call (after a code or parameter change) where I'm serializing the operation name, target shape, and fuse argument shape:
https://editor.mergely.com/sTT2EYbB

The difference is a 0 vs -0 value for one of the two shapes. I'm not sure if that's down to OC internals, and/or if I should attempt to normalize that as an edge case.

If I do, later operations have this also apparently subtle difference (0111000 vs 0101000):
https://editor.mergely.com/TVm1Cf9X

This is probably not be the goal of this PR/code, but I'm curious if you have some insight, and whether this serialization could be feasible for this use, or if the OC representation has too much built in state to make it possible

[BarbourSmith]

Interesting! I think that we are also interested in using serialized shapes for cache keys.

[tristan-huber]

Great find Billiam! Thanks for checking this out so quickly.

Our minimum requirement is logical equivalence but obviously we'd prefer absolute equivalence between serialized shapes that should be the same. The details of these serialization formats are down to OC internals so I'm not an expert in this domain.

One thing of note, both of the differences you spot are in the "triangulation" part of the serialized shape. OC has docs on the BRep Serialization format here and as far as I can tell the "triangulation" block is technically optional. It can supposedly be excluded by passing a "false" to the triangulation parameter in the more verbose implementation of BrepTools.Write. I wouldn't mind changing this replicad code to make that option available to replicad clients. That would allow you to use a triangulation-free brep as your cache key which may(?) be more stable.

That being said, I don't have a real understanding of why triangulation is default-included or what the consequences of omitting it would be. Maybe @sgenoud has some context on this? Or we could try to find someone in the OC forums.

[tristan-huber]

Getting back to this with fresh eyes. Looks like I misunderstood OC's format and while both of these diffs are near Triangulation headers they're not actually related to the triangulation content. In actuality one diff looks like it's in the Surface section and the other in the TShapes section.

Unfortunately this means it's not clear how to address this un-equalness of the serialized data. @Billiam would you mind sharing the replicad code which lead to these differences? I haven't been able to replicate on my end but I expect this issue manifests with some nuance of the specific shapes or operations you're performing.

[Billiam]

I simplified the way I was doing caching, and It may be something specific to the shapes I was working with, or the way I was caching. Here's a working example though I pulled from sgenoud's models (which works fine),
and a box generator

I'll see if I can come up with a minimal example when I can recreate it