Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deserialize boundedbtreeset #781

Merged
merged 4 commits into from
Aug 31, 2023
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 97 additions & 1 deletion bounded-collections/src/bounded_btree_set.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@
use alloc::collections::BTreeSet;
use codec::{Compact, Decode, Encode, MaxEncodedLen};
use core::{borrow::Borrow, marker::PhantomData, ops::Deref};
#[cfg(feature = "serde")]
use serde::{
de::{Error, SeqAccess, Visitor},
Deserialize, Deserializer, Serialize,
};

/// A bounded set based on a B-Tree.
///
Expand All @@ -29,9 +34,67 @@
///
/// Unlike a standard `BTreeSet`, there is an enforced upper limit to the number of items in the
/// set. All internal operations ensure this bound is respected.
#[cfg_attr(feature = "serde", derive(Serialize), serde(transparent))]
#[derive(Encode, scale_info::TypeInfo)]
#[scale_info(skip_type_params(S))]
pub struct BoundedBTreeSet<T, S>(BTreeSet<T>, PhantomData<S>);
pub struct BoundedBTreeSet<T, S>(BTreeSet<T>, #[cfg_attr(feature = "serde", serde(skip_serializing))] PhantomData<S>);

#[cfg(feature = "serde")]
impl<'de, T, S: Get<u32>> Deserialize<'de> for BoundedBTreeSet<T, S>
where
T: Ord + Deserialize<'de>,
S: Clone,
{
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
// Create a visitor to visit each element in the sequence
struct BTreeSetVisitor<T, S>(std::marker::PhantomData<(T, S)>);

impl<'de, T, S> Visitor<'de> for BTreeSetVisitor<T, S>
where
T: Ord + Deserialize<'de>,
S: Get<u32> + Clone,
{
type Value = BTreeSet<T>;

fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("a sequence")
}

fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
where
A: SeqAccess<'de>,
{
let size = seq.size_hint().unwrap_or(0);
let max = match usize::try_from(S::get()) {
Ok(n) => n,
Err(_) => return Err(A::Error::custom("can't convert to usize")),
};
if size > max {
Err(A::Error::custom("out of bounds"))
} else {
let mut values = BTreeSet::new();

while let Some(value) = seq.next_element()? {
values.insert(value);
if values.len() > max {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird. You insert before checking for the max, i.e. you add 1 more value over the limit before you check the limit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also found it weird. It was like that in BoundedVec's Deserialize implementation: https://github.com/paritytech/parity-common/blob/master/bounded-collections/src/bounded_vec.rs#L90-L92

I thought that was intentional and didn't put much thought tbh.

I can fix them both if you like?

Copy link
Contributor

@KiChjang KiChjang Aug 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but there's one more additional issue: checking values.len() > max doesn't sound right to me either, because it too is also waiting for 1 element over the max before the condition triggers. What should really happen is values.len() >= max.

It would be so great if you can write tests for these as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are already discussing the implementation of Deserialize for BoundedVec as well, I had another thing which confused me:

https://github.com/paritytech/parity-common/blob/master/bounded-collections/src/bounded_vec.rs#L84-L85

So, if we are performing a check in here for the size, do we need another check in here:
https://github.com/paritytech/parity-common/blob/master/bounded-collections/src/bounded_vec.rs#L90-L92
?

I think the second check is not necessary and the error produced by the second check is unreachable. So, I don't know if it's possible to write a test for triggering that error. It will trigger the previous error instead.

I might be missing something, so feel free to correct me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first check is using size_hint() and that could also be None in which case the size is 0 as you see in the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a new commit:

  • insertion is now performed after the bound check
  • new tests for bound check:
    • i've copied the deserializer_test and made the size equal to the bound
    • for the tests, size_hint() does indeed return None, so I'm positive I'm testing the inner check
    • if you prefer not to have an additional test, the same test can be done by changing the bound to the size of the deserialized string in the deserializer_test

return Err(A::Error::custom("out of bounds"))
}
}

Ok(values)
}
}
}

let visitor: BTreeSetVisitor<T, S> = BTreeSetVisitor(PhantomData);
deserializer
.deserialize_seq(visitor)
.map(|v| BoundedBTreeSet::<T, S>::try_from(v).map_err(|_| Error::custom("out of bounds")))?
}
}

impl<T, S> Decode for BoundedBTreeSet<T, S>
where
Expand Down Expand Up @@ -522,4 +585,37 @@
set: BoundedBTreeSet<String, ConstU32<16>>,
}
}

#[test]
fn test_serializer() {
let mut c = BoundedBTreeSet::<u32, ConstU32<6>>::new();
c.try_insert(0).unwrap();
c.try_insert(1).unwrap();
c.try_insert(2).unwrap();

assert_eq!(serde_json::json!(&c).to_string(), r#"[0,1,2]"#);

Check failure on line 596 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<6>>: serde::ser::Serialize` is not satisfied

Check failure on line 596 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest)

no method named `to_string` found for enum `Value` in the current scope

Check failure on line 596 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (macOS-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<6>>: serde::ser::Serialize` is not satisfied

Check failure on line 596 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (macOS-latest)

no method named `to_string` found for enum `Value` in the current scope
}

#[test]
fn test_deserializer() {
let c: Result<BoundedBTreeSet<u32, ConstU32<6>>, serde_json::error::Error> = serde_json::from_str(r#"[0,1,2]"#);

Check failure on line 601 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<6>>: serde::de::Deserialize<'_>` is not satisfied

Check failure on line 601 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (macOS-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<6>>: serde::de::Deserialize<'_>` is not satisfied
assert!(c.is_ok());
let c = c.unwrap();

assert_eq!(c.len(), 3);
assert!(c.contains(&0));
assert!(c.contains(&1));
assert!(c.contains(&2));
}

#[test]
fn test_deserializer_failed() {
let c: Result<BoundedBTreeSet<u32, ConstU32<4>>, serde_json::error::Error> =
serde_json::from_str(r#"[0,1,2,3,4,5]"#);

Check failure on line 614 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<4>>: serde::de::Deserialize<'_>` is not satisfied

Check failure on line 614 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (macOS-latest)

the trait bound `bounded_btree_set::BoundedBTreeSet<u32, ConstU32<4>>: serde::de::Deserialize<'_>` is not satisfied

match c {
Err(msg) => assert_eq!(msg.to_string(), "out of bounds at line 1 column 11"),

Check failure on line 617 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest)

no method named `to_string` found for struct `serde_json::Error` in the current scope

Check failure on line 617 in bounded-collections/src/bounded_btree_set.rs

View workflow job for this annotation

GitHub Actions / Test (macOS-latest)

no method named `to_string` found for struct `serde_json::Error` in the current scope
_ => unreachable!("deserializer must raise error"),
}
}
}
Loading