Marshal missing UDT fields as null instead of failing #269

sylwiaszunejko · 2024-06-05T12:09:21Z

We can't return an error in case a field is added to the UDT, otherwise existing code would break by simply altering the UDT in the database. For extra fields at the end of the UDT put nulls to be in line with gocql, but also python-driver and java-driver.

In gocql it was fixed in scylladb/gocql@d2ed1bb

java-driver https://github.com/datastax/java-driver/blob/ef56d561d97adcae48e0e6e8807f334aedc0d783/core/src/main/java/com/datastax/oss/driver/internal/core/type/codec/UdtCodec.java#L86
python-driver: https://github.com/datastax/python-driver/blob/15d715f4e686032b02ce785eca1d176d2b25e32b/cassandra/cqltypes.py#L1036

udt.go

sylwiaszunejko · 2024-06-10T06:04:01Z

@avelanarius ping

dkropachev

We definitely should have test for that

dkropachev · 2024-06-14T11:53:50Z

udt.go

@@ -39,11 +39,17 @@ func makeUDT(value reflect.Value, mapper *reflectx.Mapper, unsafe bool) udt {

 func (u udt) MarshalUDT(name string, info gocql.TypeInfo) ([]byte, error) {
 	value, ok := u.field[name]
-	if !ok {
-		return nil, fmt.Errorf("missing name %q in %s", name, u.value.Type())
+


Should we consider unsafe flag ?

func (u udt) MarshalUDT(name string, info gocql.TypeInfo) ([]byte, error) { value, ok := u.field[name] if ok { return gocql.Marshal(info, value.Interface()) } if u.unsafe { return nil, nil } return nil, fmt.Errorf("missing name %q in %s", name, u.value.Type()) }

@dkropachev AFAIK this would require a user to use query.Iter().Unsafe() instead of just query.Exec(), because in gocql implementation there is no unsafe equivalent. Wouldn't it be counterintuitive if in gocql we are able to do query.Exec() with udt with missing fields and it works, but in gocqlx we are required to use Unsafe()?

@sylwiaszunejko , 100% agree on that, but since Unsafe is there and does control this particular behavior we have to be consistent in the way it works.

sylwiaszunejko · 2024-06-14T15:56:04Z

To sum up after a discussion with @dkropachev:
There is a mechanism in gocqlx for ignoring missing UDT fields, but it requires a user to use query.Iter().Unsafe() instead of just query.Exec(), because in gocql implementation there is no Unsafe equivalent. So on one hand Unsafe is there and does control this particular behavior we have to be consistent in the way it works, but on the other hand we have to be consistent with gocql and if using UDTs with extra fields is supported there always not only when extra option specified, shouldn't it be also supported in gocqlx?
@mykaul do you have any thoughts on that? Because it looks like my fix doesn't take Unsafe mechanism in the consideration and that's probably not the best

dkropachev · 2024-06-14T16:02:55Z

To sum up after a discussion with @dkropachev: There is a mechanism in gocqlx for ignoring missing UDT fields, but it requires a user to use query.Iter().Unsafe() instead of just query.Exec(), because in gocql implementation there is no Unsafe equivalent. So on one hand Unsafe is there and does control this particular behavior we have to be consistent in the way it works, but on the other hand we have to be consistent with gocql and if using UDTs with extra fields is supported there always not only when extra option specified, shouldn't it be also supported in gocqlx? @mykaul do you have any thoughts on that? Because it looks like my fix doesn't take Unsafe mechanism in the consideration and that's probably not the best

I would propose to rename this attribute to Strict reversing it's meaning, making default to be false.
Only problem here is that we have to release new major for that, due to the compatibility issue.

mykaul · 2024-06-14T16:05:05Z

I'm in favor of consistency, even if it's the longer path (and complexity). The unsafe is not intuitive.

dkropachev · 2024-06-14T16:14:35Z

@mykaul , could you please rereview it, tests were added and more cases fixed.
Also new method was introduced to Queryx to control Unsafe flag on it's level instead of borrowing it from DefaultUnsafe

mykaul · 2024-06-18T06:03:27Z

Do we have an example (in gocqlx docs) for UDT usage?

sylwiaszunejko · 2024-06-18T08:25:15Z

@mykaul examples are here:

gocqlx/doc_test.go

Line 24 in a9ab270

func ExampleUDT() {

and

gocqlx/example_test.go

Line 445 in a9ab270

// This example shows how to add User Defined Type marshalling capabilities to

wprzytula · 2024-06-19T08:09:37Z

iterx.go

+//   - ptr to t implements gocql.Unmarshaler, gocql.UDTUnmarshaler or UDT
+//   - it is not a struct
+//   - it has no exported fields


Why this change? Asterisks are valid Markdown itemizers as well AFAIK.

This one is gone, it is already in master, reason why it have changed is golangci-lint does not tollerate anything by - there.

iterx_test.go

queryx.go

wprzytula · 2024-06-19T08:20:44Z

udt.go

 func (u udt) UnmarshalUDT(name string, info gocql.TypeInfo, data []byte) error {
 	value, ok := u.field[name]
-	if !ok && !u.unsafe {
-		return fmt.Errorf("missing name %q in %s", name, u.value.Type())
+	if ok {
+		return gocql.Unmarshal(info, data, value.Addr().Interface())
 	}
-
-	return gocql.Unmarshal(info, data, value.Addr().Interface())
+	if u.unsafe {
+		return nil
+	}
+	return fmt.Errorf("missing name %q in %s", name, u.value.Type())


Are we sure this is the semantics of unsafe that we want to have?
In Rust driver at least, we consider too many and too few fields in the CQL UDT compared to the go struct as two distinct cases. In Rust driver, strict forbids too many fields in CQL UDT, whereas default_if_missing is used for the case when a struct has a field that is not found in the CQL UDT metadata. WDYT? Should we cover it by a single lever here?

@wprzytula, i am totally in support of having all drivers as close as possible to each other. Let's have separate issue on that and address it in other PR

OTOH I have no idea how other drivers apart from Rust driver approach this.

We can start from these two

wprzytula · 2024-06-19T08:21:37Z

I'm strongly for the change from unsafe to strict, and making non-strict the default. This would be a breaking change, but one that I find justified.

sylwiaszunejko · 2024-06-19T08:28:04Z

I'm strongly for the change from unsafe to strict, and making non-strict the default. This would be a breaking change, but one that I find justified.

I agree, I would like to go with the strict approach.

@mykaul @avelanarius @roydahan Are we okay with making breaking change in that case?

We can't return an error in case a field is added to the UDT, otherwise existing code would break by simply altering the UDT in the database. For extra fields at the end of the UDT put nulls to be in line with gocql, but also python-driver and java-driver. In gocql it was fixed in scylladb/gocql@d2ed1bb

It enables local control over `unsafe` mode for .Bind methods of `Queryx` and iterators spawn by it.

…safe mode We can't return an error in case a field is added to the UDT, otherwise existing code would break by simply altering the UDT in the database. For extra fields at the end of the UDT put nulls to be in line with gocql, but also python-driver and java-driver. In gocql it was fixed in scylladb/gocql@d2ed1bb

dkropachev · 2024-06-19T17:30:22Z

I'm strongly for the change from unsafe to strict, and making non-strict the default. This would be a breaking change, but one that I find justified.

I agree, I would like to go with the strict approach.

@mykaul @avelanarius @roydahan Are we okay with making breaking change in that case?

Let's do that in separate PR.

roydahan · 2024-06-23T23:04:31Z

I'm fine with it, we can release a major and state that in the release notes.

sylwiaszunejko · 2024-06-25T10:29:29Z

Let's merge it and then we can replace unsafe with strict approach in the additional PR. Are you okay with that?
@wprzytula @roydahan @mykaul

mmatczuk · 2024-06-25T14:15:17Z

Interesting.

mmatczuk · 2024-06-25T14:21:22Z

I'd refrain from bumping the major version and only do that in case of major API breakage i.e. new API, preferably never.

Bumping it to v2 was a complete mess, every import line needs to be changed.
Not that it's hard, but it puts quite a lot of annoying work on customers.

sylwiaszunejko · 2024-06-25T14:23:49Z

@mmatczuk do you have any suggestion how to handle better the situation with Unsafe and how the default behavior around dealing with missing fields in UDTs differs in gocql and gocqlx?

mmatczuk · 2024-06-25T14:31:56Z

The change is LGTM.

sylwiaszunejko · 2024-06-25T14:49:02Z

I am not sure if I understand right, is it okay to do breaking change without major release?

mmatczuk · 2024-06-25T15:01:58Z

What I'm trying to say is that there is a cost of making /v3 it is not something to do lightly.
Consider what you need, maybe completely removing unsafe and releasing /v3 is a way to go.

dkropachev · 2024-06-26T16:23:49Z

What I'm trying to say is that there is a cost of making /v3 it is not something to do lightly. Consider what you need, maybe completely removing unsafe and releasing /v3 is a way to go.

Just want to point out that gocqlx recently switched to scylladb/gocql, which also contributes to necessity to bump major up.

sylwiaszunejko requested a review from avelanarius June 5, 2024 12:12

sylwiaszunejko assigned mykaul and sylwiaszunejko and unassigned mykaul Jun 5, 2024

sylwiaszunejko requested a review from mykaul June 5, 2024 12:13

mykaul reviewed Jun 5, 2024

View reviewed changes

udt.go Outdated Show resolved Hide resolved

sylwiaszunejko force-pushed the missing_udt_fields branch from 3ef9792 to bdee62b Compare June 5, 2024 12:45

roydahan requested a review from dkropachev June 13, 2024 23:45

dkropachev reviewed Jun 14, 2024

View reviewed changes

sylwiaszunejko requested a review from mykaul June 14, 2024 15:56

dkropachev force-pushed the missing_udt_fields branch from bdee62b to 31e64f5 Compare June 14, 2024 16:04

dkropachev force-pushed the missing_udt_fields branch from 31e64f5 to 4d8c8ba Compare June 14, 2024 16:12

dkropachev force-pushed the missing_udt_fields branch from 4d8c8ba to 2239c1b Compare June 14, 2024 16:58

wprzytula reviewed Jun 19, 2024

View reviewed changes

sylwiaszunejko and others added 2 commits June 19, 2024 08:37

Introduce Unsafe method on Queryx

8238a44

It enables local control over `unsafe` mode for .Bind methods of `Queryx` and iterators spawn by it.

dkropachev force-pushed the missing_udt_fields branch from 2239c1b to cdb2e11 Compare June 19, 2024 12:38

dkropachev force-pushed the missing_udt_fields branch from cdb2e11 to a12aa3e Compare June 19, 2024 12:50

sylwiaszunejko closed this Jun 25, 2024

sylwiaszunejko reopened this Jun 25, 2024

sylwiaszunejko requested a review from wprzytula June 25, 2024 10:29

wprzytula approved these changes Jun 25, 2024

View reviewed changes

sylwiaszunejko merged commit 6a60650 into scylladb:master Jun 25, 2024
2 checks passed

sylwiaszunejko mentioned this pull request Jun 25, 2024

Replace Unsafe with Strict mechanism #276

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Marshal missing UDT fields as null instead of failing #269

Marshal missing UDT fields as null instead of failing #269

sylwiaszunejko commented Jun 5, 2024

sylwiaszunejko commented Jun 10, 2024

dkropachev left a comment

dkropachev Jun 14, 2024

sylwiaszunejko Jun 14, 2024

dkropachev Jun 14, 2024

sylwiaszunejko commented Jun 14, 2024

dkropachev commented Jun 14, 2024

mykaul commented Jun 14, 2024

dkropachev commented Jun 14, 2024

mykaul commented Jun 18, 2024

sylwiaszunejko commented Jun 18, 2024

wprzytula Jun 19, 2024

dkropachev Jun 19, 2024

wprzytula Jun 19, 2024

dkropachev Jun 19, 2024

wprzytula Jun 19, 2024

dkropachev Jun 19, 2024

wprzytula commented Jun 19, 2024

sylwiaszunejko commented Jun 19, 2024

dkropachev commented Jun 19, 2024

roydahan commented Jun 23, 2024

sylwiaszunejko commented Jun 25, 2024 •

edited

Loading

mmatczuk commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

sylwiaszunejko commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

sylwiaszunejko commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

dkropachev commented Jun 26, 2024 •

edited

Loading

Marshal missing UDT fields as null instead of failing #269

Marshal missing UDT fields as null instead of failing #269

Conversation

sylwiaszunejko commented Jun 5, 2024

sylwiaszunejko commented Jun 10, 2024

dkropachev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sylwiaszunejko commented Jun 14, 2024

dkropachev commented Jun 14, 2024

mykaul commented Jun 14, 2024

dkropachev commented Jun 14, 2024

mykaul commented Jun 18, 2024

sylwiaszunejko commented Jun 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wprzytula commented Jun 19, 2024

sylwiaszunejko commented Jun 19, 2024

dkropachev commented Jun 19, 2024

roydahan commented Jun 23, 2024

sylwiaszunejko commented Jun 25, 2024 • edited Loading

mmatczuk commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

sylwiaszunejko commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

sylwiaszunejko commented Jun 25, 2024

mmatczuk commented Jun 25, 2024

dkropachev commented Jun 26, 2024 • edited Loading

sylwiaszunejko commented Jun 25, 2024 •

edited

Loading

dkropachev commented Jun 26, 2024 •

edited

Loading