-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding support for Exact Numeric Types #10190
Comments
We have an internal custom aggregator based on BigDecimal which is being used in several production usecases. We'll look into opensourcing it for the benefit of the community. |
@a2l007 that sounds great! I have been looking at this area recently too, and wondering if there is any performance issue with using BigDecimal. Do you have know how slow it is compared to aggregators based on primitive types? |
We use a variant of BigDecimal called CompressedBigDecimal with the difference being that the latter is mutable and instead of using the object representation for complex types, it is represented as block compressible columns. Haven't had a chance to do performance testing against the primitive types, but the CompressedBigDecimal based aggregator performs significantly better than BigDecimal itself. |
Oh I see. Looking forward to seeing it! |
extension that @a2l007 mentioned (based on what I recall and work done when I was at Yahoo): FWIW, I also did a prototype using java BigDecimal data type that was too slow for the use case, also tried creating a custom BigDecimal data type using two longs (one for integer part and other for fractional part) which also couldn't match the performance obtained by simpler approach above. |
@himanshug thanks for sharing. I was thinking the same. Did the performance not meet only your requirement or just slow in general? |
@jihoonson java's One line of thought was to create a custom |
@himanshug oh sorry, in your last comment, I missed that the performance wasn't as good as the simpler approach 😅 I'm considering adding a new |
FYI the proposal for the compressed bigdecimal extension is up #10668 |
This issue has been marked as stale due to 280 days of inactivity. |
Hi, I'm working on an application which deals with aggregating a few currency-based fields and was experimenting with using this data type extension. However, part of our application requires using Druid SQL queries, which seem to not support Compressed BigDecimal. I understand native queries are capable of performing aggregation functions, however is there any plan to support Druid SQL queries? Specifically I'm trying to use the summation of certain columns, however Druid SQL's SUM operator does not seem to support Complex data types. |
This issue has been marked as stale due to 280 days of inactivity. |
Description
Druid supports Double data type which is an Approximate Numeric Types. Is it on the roadmap anywhere to add support for Exact Numeric Types?
Highest numerical precision we can get with druid is the double (64bit) data type. Due to the nature of binary storage format, we would have 53 meaningful bits, which means 15 meaningful decimal digits for sure. ( 53 log10(2) ≈ 15.955 ).
As an example of Exact Numeric type: BigQuery has support for NUMERIC data type which gives 38 digits of precision and 9 decimal digits of scale. This is the ideal field for financial reporting, or any high precision calculations.
Motivation
Support for data with High precision ranging that contains very small and very big data points without data loss.
The text was updated successfully, but these errors were encountered: