Consider merging with Serde

[erasmospunk]

The Serde library supports many serialization formats but not capnproto.

One useful feature is the transcode functionality where it is possible to convert from one format to the other.

[dwrensha]

Hi!

My impression is that Cap'n Proto would not be a very good fit for Serde integration, mainly because Cap'n Proto wants to define data types in schemas external to Rust code. I have an old post on this topic. (The libserialize discussed there is now called rustc_serialize, if I remember correctly.)

Do you have a specific use case in mind?

[burdges]

I'd think cap'n proto could generate the serde attributes from its schema, maybe useful if someone must work around company policies like always use json or xml for such and such.

[erasmospunk]

Good point about the schema, I initially thought that capnpc-rust could generate the needed structs with Serde's annotations & any runtime code. Not sure if this is simple, especially considering that Serde does not support protocol buffers.

My use case is converting from JSON to Capnproto as an internal representation for microservices. Using the Serde lib could potentially make the transcoding with less code.

If you think that this is something that does not make sense for this project, feel free to close this issue.

[dwrensha]

I do think think that it would be good to have JSON <--> Cap'n Proto support in capnproto-rust, but I'm not sure that it would make sense to involve Serde. I would probably start by looking at how capnproto-c++ supports JSON encoding and decoding.

[erasmospunk]

Thanks for feedback, closing this issue...

[dwrensha]

Today I came across serde-protobuf. I'm not sure exactly what it does, but it seems relevant to this discussion.

[piaoger]

Do you have a plan to support JSON encoding and decoding in capnproto-rust?

[dwrensha]

@piaoger It would be good to have, but I don't have any immediate plans to work on it.

I think the way forward would be to implement dynamic reflection as in the C++ reference implementation, so that to_json() and from_json() functions could work for any type of message without needing any extra generated code.

[kaligrafy]

Any news on this?

[sffc]

Cap'n Proto works with aligned data, so it has the capability to return things like &[u32] on little-endian systems, which can be crucial for performance.

Serde on the other hand is capable of zero-copy deserializing &[u8] (and &str) but not anything wider than that. You would therefore be losing out on certain performance characteristics and zero-copy deserialization by trying to fit Cap'n Proto into Serde's data model.

It is possible to model slices of complex types on top of byte buffers by using unaligned reads, which incurs a small performance penalty, but in practice this penalty is often in the noise. The zerovec crate provides these abstractions so that you can use zero-copy deserialization of complex types with Serde. (Disclaimer: I am a maintainer of the zerovec crate.)

[burdges]

As an aside, we discussed multi- architecture layout ideas in https://internals.rust-lang.org/t/multi-architecture-layout/18538 which likely never happens in rustc, but maybe gives someone ideas.

[puremourning]

@piaoger It would be good to have, but I don't have any immediate plans to work on it.

I think the way forward would be to implement dynamic reflection as in the C++ reference implementation, so that to_json() and from_json() functions could work for any type of message without needing any extra generated code.

I had a use-case for serialising cap'n'proto messages to JSON and back, and noticed that the rust implementation doesn't have the same JSON codec as the C++ implementation. So I started work on a compatible rust implementation; I made it so that the serialisation and annotations are equivalent to and compatible with the C++ implementation in compat/json.capnp. it of course uses the dynamic reflection that's now supported in the rust implementation of capnproto.

It's just a POC right now, but @kentonv would this be something that's worth progressing to a full PR?

[dwrensha]

Can you elaborate on how the current capnproto-rust json codec differs from the C++ version?

[dwrensha]

Oh, right -- there isn't actually a json codec in the main capnproto-rust repo right now.