Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve error message with recursive normalisers #1299

Closed
DrNickClarke opened this issue Feb 5, 2024 · 2 comments
Closed

Improve error message with recursive normalisers #1299

DrNickClarke opened this issue Feb 5, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@DrNickClarke
Copy link
Collaborator

DrNickClarke commented Feb 5, 2024

Describe the bug

When large/complex structures are stored using recursive normalisers, it fails with a confusing error that indicates that user metadata is too large. Even when the user has not specified metadata.

Steps/Code to Reproduce

N = 100_000
d = {}
for i in range(N):
    k = f"key_{i}"
    v = pd.DataFrame({'col': np.arange(float(i), i + 100.)})
    d[k] = v
lib.write_pickle('dict', d)

Takes a long time on the write_pickle but eventually completes with a warning

[2024-07-02 14:12:27.493] [arcticdb] [warning] User defined metadata is above warning size (8388608B), metadata cannot exceed 16777216B.  Current size: 15489024B.

Expected Results

The error message should state that the structure being normalised is too large/complex.

OS, Python Version and ArcticDB Version

adb 4.0.3

Backend storage used

On-prem S3

Additional Context

No response

@DrNickClarke DrNickClarke added the bug Something isn't working label Feb 5, 2024
@jamesmunro
Copy link
Collaborator

@DrNickClarke Can you add steps to generate the misleading error message and a proposed improvements please?

@DrNickClarke
Copy link
Collaborator Author

Closed after moving to the internal repo here https://github.com/man-group/arcticdb-man/issues/127

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants