Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(Buffer): uint32 -> uint64 in slice methods #323

Merged
merged 1 commit into from
Sep 1, 2023

Conversation

mocurin
Copy link
Contributor

@mocurin mocurin commented Dec 21, 2022

Problem

I'm currently on Dgraph 21.12 unable to export my data.
Export fails on just two nodes out of 7 with the "Unexpected EOF" error:

Dec 14 11:31:51 dm-dgraph-04 dgraph[224355]: I1214 11:31:51.840130  224355 log.go:34] Export [01h26m22s] Scan (12): ~2.1 TiB/2.5 TiB at 177 MiB/sec. Sent: 801.5 GiB at 231 MiB/sec. jemalloc: 7.5 GiB
Dec 14 11:31:55 dm-dgraph-04 dgraph[224355]: W1214 11:31:55.408201  224355 log.go:36] Error while sending: unexpected EOF

Skipping rather long investigation of this issue I came to find length of slice, written to the Buffer during export exceed the size of uint32 (i've decoded varint before Value field in Badger KV struct with RDF's to get something around 4.5Gb, which is expected for a rather bloated reverse edge to the one of the most common nodes in my DB. Also count query returns 72 105 794 connected nodes which is, welp, quite a lot).

Not to mention that working with int which is almost always is int64 and then casually casting it to uint32 w/o any checks or warnings is as bad as it gets.

Solution

Find any 4 and Uint32 and carefully replace them with 8 and Uint64. As this happens only in slice-related methods the fix is quite easy. Locally tests run just fine, but i had to patch the sort one to accommodate for size changes. Also i did test 21.12-related badger version and tests run fine too.

Afterword

I'm somewhat in a hurry, so sorry for not being able to provide more info. I'll write something here some time later.
Also, as far as a can tell - any version of dgraph should be affected by this. Not sure how I turned out to be the first one.

@CLAassistant
Copy link

CLAassistant commented Dec 21, 2022

CLA assistant check
All committers have signed the CLA.

@mocurin
Copy link
Contributor Author

mocurin commented Dec 25, 2022

Seems that coveralls report posting failed

@mangalaman93 mangalaman93 merged commit c00b352 into dgraph-io:main Sep 1, 2023
5 checks passed
@mangalaman93
Copy link
Contributor

Thank you for the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants