Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cosmos-sdk-proto: add serde derive macros #471

Merged
merged 24 commits into from
Aug 13, 2024

Conversation

ash-burnt
Copy link
Contributor

@ash-burnt ash-burnt commented Jun 1, 2024

Hello,

I am currently trying to use cosmos protos for storing information inside of cosmwasm smart contracts, but the existing definitions don't have the required Serialize and Deserialize traits. This should fix #409 and #83

I have extended the proto-build buf configuration to run an additional neoeinstein plugin to generate these traits.

@romac
Copy link
Member

romac commented Jul 29, 2024

Be warned that pbjson is not no_std compatible if I recall correctly. In ics23 and in ibc-proto we use our own fork of pbjson with no_std compatibility: https://github.com/informalsystems/pbjson

@tony-iqlusion
Copy link
Member

@romac whatever is used to solve informalsystems/tendermint-rs#1445 we can use here too

@tony-iqlusion
Copy link
Member

@ash-burnt now that #482 is merged, if you rebase there should be a way forward here

tendermint-proto = "0.39"
serde = "1.0.203"
pbjson = { git = "https://github.com/informalsystems/pbjson.git", package = "informalsystems-pbjson" }
pbjson-types = { git = "https://github.com/informalsystems/pbjson.git", package = "informalsystems-pbjson-types" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, the extern_paths have been updated to point to the new tendermint rs version, and these deps have been removed

Comment on lines 23 to 29
const COSMOS_SDK_REV: &str = "v0.47.10";

/// The Cosmos ibc-go commit or tag to be cloned and used to build the proto files
const IBC_REV: &str = "v3.0.0";

/// The wasmd commit or tag to be cloned and used to build the proto files
const WASMD_REV: &str = "v0.29.2";
const WASMD_REV: &str = "v0.45.0";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you bump these in a separate PR? There's already quite a bit going on in this one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have rolled back the version updates

pub config: ::core::option::Option<::tendermint_proto::google::protobuf::Any>,
pub config: ::core::option::Option<::pbjson_types::Any>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should stay as they are. Instead of using pbjson-types, you can leverage the newly added pbjson support in tendermint-proto (which solves every downstream crate having to figure out this problem)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After updating the extern_paths, these now generate as they did before

@tony-iqlusion
Copy link
Member

@ash-burnt looks like there are conflicts that need resolved

@ash-burnt
Copy link
Contributor Author

@tony-iqlusion they should be addressed now

@ash-burnt
Copy link
Contributor Author

Looks like these serde generations still require pbjson for serialization and deserialization of non-proto values. Seems like numbers and binary types

@tony-iqlusion
Copy link
Member

@ash-burnt yes, the (informalsystems-)pbjson crate is fine, it's just pbjson-types we're trying to avoid in lieu of using the definitions in tendermint-proto

Co-authored-by: Tony Arcieri (iqlusion) <tony@iqlusion.io>
Co-authored-by: Tony Arcieri (iqlusion) <tony@iqlusion.io>
@tony-iqlusion
Copy link
Member

Aah, so the problem isn't serde itself, but no_std in tendermint-proto

I think we're going to need to add a feature for this and have it enable the std feature of tendermint-proto.

Either that, or we're going to need to add a set of imports to the top of each of the generated .serde.rs files in lieu of the prelude.

@ash-burnt
Copy link
Contributor Author

What should the feature be called? serde?

@tony-iqlusion
Copy link
Member

Sure, that's fine

@ash-burnt
Copy link
Contributor Author

Went with "serialization" since it can't conflict with an import name


[features]
default = ["grpc-transport"]
std = ["prost/std", "tendermint-proto/std"]
grpc = ["std", "tonic"]
grpc-transport = ["grpc", "tonic/transport"]
cosmwasm = []
serialization = ["serde", "tendermint-proto/std", "pbjson"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot of serialization going on in this crate, both protobuf and now JSON.

How about:

Suggested change
serialization = ["serde", "tendermint-proto/std", "pbjson"]
serde = ["dep:serde", "tendermint-proto/std", "pbjson"]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, and adjusted the code gen

@tony-iqlusion tony-iqlusion changed the title Add Serde derive macros cosmos-sdk-proto: add serde derive macros Aug 13, 2024
@tony-iqlusion
Copy link
Member

@ash-burnt needs rustfmt but otherwise looks good

@ash-burnt
Copy link
Contributor Author

@tony-iqlusion formatted, let me know if there is anything else that needs doing

Comment on lines +382 to +392
// add the feature flag to the serde definitions
(
"impl serde::Serialize for",
"#[cfg(feature = \"serde\")]\n\
impl serde::Serialize for",
),
(
"impl<'de> serde::Deserialize<'de> for",
"#[cfg(feature = \"serde\")]\n\
impl<'de> serde::Deserialize<'de> for",
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These can probably be gated on the module level but I don't see any reason to block the PR over that

@tony-iqlusion tony-iqlusion merged commit 7e55c1d into cosmos:main Aug 13, 2024
12 checks passed
This was referenced Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add additional derive macros to Prost generated types
3 participants