Replace protobuf messages with IPLD

[chafey]

go-graphsync currently uses google protobuf for messaging. Since graphsync depends on IPLD and IPLD can be used in place of GPB, we can reduce dependencies, tooling and simplify the design by replacing GPB with IPLD. IPLD schema can be used to define the messages and dag-cbor could be used to encode them. @warpfork said this was previously discussed but cut due to time constraints - adding this issue for tracking

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[hannahhoward]

I am fully on board with this but would like it to be done in a backwards compatible way, as Filecoin mainnet is likely to ship with protobuf graphsync

In other words, I'd like to keep the protobuf protocol and define a new versioned protocol for the IPLD version, and gradually phase out the old one.

Also, serialization/deserialization of messages is very speed and memory critical, so we may need a code generated node or other specialized IPLD type to make it work.

I started down this before, and hit some challenges, that I want to lay out here -- not to discourage, but basically to cover all the dragons ahead of time -- I think getting these messages encoded/decoded with ipld-prime is a GREAT use case for proving IPLD primes performance abilities and flexibility.

---

I hit a bit of a barrier last time around the extensions. In the current protobuff work they are:

map[string][]byte

But actually, the []byte is just CBOR-encoded IPLD data. Once the whole thing is a CBOR IPLD struct, by IPLD schema is effectively:

type ExtensionName string
type Extensions {ExtensionName:Any}
type Request {
   ...
   Extensions Extensions
   ...
}

Except last I had checked there was still contention about whether IPLD schemas included an Any type.

Moreover, because there's no obvious way to fast deserialize an "Any" type of IPLD data, I had planned to defer deserialziation (note, this is different than a simply []byte type -- deferring deserialization means that the overall encoded data structure is still a correct CBOR encoding of the nested data).

Side note: After hitting this barrier, I had planned for my first quick and dirty version to not try to use ipld-prime at all and rather just use https://github.com/whyrusleeping/cbor-gen along with the Deferred type it uses for holding of on deserialization of certain types

One other challenge: Selectors -- in the protobuf we write them as []byte, but obviously they are just a node embedded in the larger structure. We don't currently have a custom node type for selectors. A Selector is an IPLD Node, but since we don't have a custom node type, we read them in as generic nodes and run parsing code to convert them into an executable Selector. I believe @warpfork's intent was to eventually just have a custom type, which makes sense to me. Then you can properly serialize/deserialize into the message and make it executable in a single step.

Anyway, these are the things that made me go ... "oh better go jump on some filecoin challenges"

[chafey]

Thanks for this information, it is really helpful! I will maintain backwards compatibility with the existing probobuf based version.

[hannahhoward]

see for Proposal on format to use