Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Field aggregation #8786

Merged
merged 48 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
bcabc13
feat: basic ui for aggregation
DarkPhoenix2704 Jun 24, 2024
b6fec06
feat: update aggregation in ui
DarkPhoenix2704 Jun 24, 2024
bb10b80
feat: aggregation api implementation
DarkPhoenix2704 Jun 24, 2024
5b9c33c
feat: attachment aggregation.ts
DarkPhoenix2704 Jun 24, 2024
ead920d
fix: some changes
DarkPhoenix2704 Jun 24, 2024
0b40d9f
fix: rebase
DarkPhoenix2704 Jun 24, 2024
a6ad370
feat: aggregation for links, rollup, ltar, formula, lookup
DarkPhoenix2704 Jun 24, 2024
5e38aa7
fix: type errors
DarkPhoenix2704 Jun 24, 2024
5002364
fix: move from data-alias controller, service to data-table service, …
DarkPhoenix2704 Jun 24, 2024
d13cd56
chore: inline docs for aggregations
DarkPhoenix2704 Jun 24, 2024
60cbe48
fix: handle edge cases
DarkPhoenix2704 Jun 24, 2024
11ea86a
fix: ui bugs
DarkPhoenix2704 Jun 24, 2024
ef99e1f
feat: working ui aggregation
DarkPhoenix2704 Jun 24, 2024
9785ae2
fix: minor issue
DarkPhoenix2704 Jun 24, 2024
88fc58a
fix: rollup and links fix count
DarkPhoenix2704 Jun 24, 2024
4db81c9
fix: handle ID Column
DarkPhoenix2704 Jun 24, 2024
6e3f58a
fix: minor fixes
DarkPhoenix2704 Jun 24, 2024
505f1d6
fix: update aggregation on data change
DarkPhoenix2704 Jun 24, 2024
c30cbf2
fix: round to 2 decimal places
DarkPhoenix2704 Jun 24, 2024
4f207ad
fix: stddev computation error replace with stddev_pop
DarkPhoenix2704 Jun 24, 2024
6c472e7
fix: use pg age function
DarkPhoenix2704 Jun 24, 2024
4394aed
feat: new record layout
DarkPhoenix2704 Jun 24, 2024
4dcacac
fix: shared view aggregations
DarkPhoenix2704 Jun 24, 2024
810ab30
feat: aggregations based on formula result
DarkPhoenix2704 Jun 24, 2024
bbc5253
fix: temp pagination
DarkPhoenix2704 Jun 24, 2024
8a67102
feat: ncpagination v2
DarkPhoenix2704 Jun 24, 2024
6d7cfe3
feat: ncpagination v2
DarkPhoenix2704 Jun 24, 2024
4ed34d1
fix: playwright tests
DarkPhoenix2704 Jun 24, 2024
a7c5511
fix: pending changes
DarkPhoenix2704 Jun 24, 2024
f60987b
fix: failing tests
DarkPhoenix2704 Jun 24, 2024
cb63ba9
feat: mysql2 aggregations
DarkPhoenix2704 Jun 24, 2024
e518aa9
fix: build
DarkPhoenix2704 Jun 24, 2024
180b2be
fix: record count
DarkPhoenix2704 Jun 24, 2024
3af7de0
fix: cleanup
DarkPhoenix2704 Jun 24, 2024
444db0a
fix: disable count aggregation
DarkPhoenix2704 Jun 24, 2024
e7f439a
feat: expiremental sqlite3 aggregation
DarkPhoenix2704 Jun 24, 2024
3b32bd4
fix: mysql2 median
DarkPhoenix2704 Jun 24, 2024
c90074e
fix:minor issues
DarkPhoenix2704 Jun 24, 2024
300f1f9
refactor: rename column to column_query fix: remove default aggregati…
DarkPhoenix2704 Jun 24, 2024
f64bc41
fix: remove unwanted else case
DarkPhoenix2704 Jun 24, 2024
cbfcea8
fix: aggregation not loading
DarkPhoenix2704 Jun 24, 2024
3cdb85e
fix: rebase
DarkPhoenix2704 Jun 24, 2024
9665b94
fix: rebase
DarkPhoenix2704 Jun 24, 2024
9370a68
fix: pagination fixed height
DarkPhoenix2704 Jun 24, 2024
d0e2a36
fix: respect locked mode for aggregations
DarkPhoenix2704 Jun 24, 2024
2840a7f
fix: pagination component
DarkPhoenix2704 Jun 24, 2024
cd4ba54
fix: pagination component
DarkPhoenix2704 Jun 24, 2024
50f6ae2
fix: replace Math.random
DarkPhoenix2704 Jun 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: aggregations based on formula result
  • Loading branch information
DarkPhoenix2704 committed Jun 24, 2024
commit 810ab304e47d576076b40108753b78aa41b868cc
8 changes: 2 additions & 6 deletions packages/nc-gui/components/smartsheet/grid/PaginationV2.vue
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ onMounted(() => {
<template #overlay>
<NcMenu>
<NcMenuItem
v-for="(agg, index) in getAggregations(displayFieldComputed.column.uidt, true)"
v-for="(agg, index) in getAggregations(displayFieldComputed.column, true)"
:key="index"
@click="updateAggregate(displayFieldComputed.column.id, agg)"
>
Expand Down Expand Up @@ -123,11 +123,7 @@ onMounted(() => {

<template #overlay>
<NcMenu>
<NcMenuItem
v-for="(agg, index) in getAggregations(column.uidt)"
:key="index"
@click="updateAggregate(column.id, agg)"
>
<NcMenuItem v-for="(agg, index) in getAggregations(column)" :key="index" @click="updateAggregate(column.id, agg)">
<div class="flex !w-full text-gray-800 items-center justify-between">
{{ $t(`aggregation_type.${agg}`) }}

Expand Down
12 changes: 9 additions & 3 deletions packages/nc-gui/composables/useViewAggregate.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { Ref } from 'vue'
import { CommonAggregations, type TableType, type ViewType, getAvailableAggregations } from 'nocodb-sdk'
import { type ColumnType, CommonAggregations, type TableType, UITypes, type ViewType, getAvailableAggregations } from 'nocodb-sdk'

const [useProvideViewAggregate, useViewAggregate] = useInjectionState(
(
Expand Down Expand Up @@ -56,8 +56,14 @@ const [useProvideViewAggregate, useViewAggregate] = useInjectionState(
}
})

const getAggregations = (type: string, hideNone?: boolean) => {
const agg = getAvailableAggregations(type)
const getAggregations = (column: ColumnType, hideNone?: boolean) => {
let agg
if (column.uidt === UITypes.Formula && (column.colOptions as any)?.parsed_tree?.dataType) {
agg = getAvailableAggregations(column.uidt!, (column.colOptions as any).parsed_tree)
} else {
agg = getAvailableAggregations(column.uidt!)
}

if (hideNone) {
return agg.filter((x) => x !== 'none')
}
Expand Down
22 changes: 21 additions & 1 deletion packages/nocodb-sdk/src/lib/aggregationHelper.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import UITypes from './UITypes';
import { FormulaDataTypes } from '~/lib/formulaHelpers';

enum NumericalAggregations {
Sum = 'sum',
Expand Down Expand Up @@ -48,7 +49,26 @@ const AllAggregations = {
...DateAggregations,
};

const getAvailableAggregations = (type: string): string[] => {
const getAvailableAggregations = (type: string, parsed_tree?): string[] => {
if (type === UITypes.Formula && parsed_tree?.dataType) {
switch (parsed_tree.dataType) {
case FormulaDataTypes.BOOLEAN:
return [...Object.values(BooleanAggregations), CommonAggregations.None];
case FormulaDataTypes.DATE:
return [
...Object.values(DateAggregations),
...Object.values(CommonAggregations),
];
case FormulaDataTypes.NUMERIC:
return [
...Object.values(NumericalAggregations),
...Object.values(CommonAggregations),
];
default:
return [...Object.values(CommonAggregations)];
}
}

switch (type) {
case UITypes.Number:
case UITypes.Decimal:
Expand Down
34 changes: 27 additions & 7 deletions packages/nocodb/src/db/aggregation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
BooleanAggregations,
CommonAggregations,
DateAggregations,
FormulaDataTypes,
getAvailableAggregations,
NumericalAggregations,
UITypes,
Expand All @@ -16,7 +17,10 @@ import genRollupSelectv2 from '~/db/genRollupSelectv2';
import generateLookupSelectQuery from '~/db/generateLookupSelectQuery';

const validateColType = (column: Column, aggregation: string) => {
const agg = getAvailableAggregations(column.uidt);
const agg = getAvailableAggregations(
column.uidt,
column.colOptions?.parsed_tree,
);

if (!agg.includes(aggregation)) {
NcError.badRequest(
Expand Down Expand Up @@ -122,6 +126,8 @@ export default async function applyAggregation({
if (_column.uidt === UITypes.LastModifiedBy && !_column.column_name)
column = 'updated_by';

const parsedFormulaType = _column.colOptions?.parsed_tree?.dataType;

/* The following column types require special handling for aggregation:
* - Links
* - Rollup
Expand Down Expand Up @@ -161,7 +167,6 @@ export default async function applyAggregation({
break;
}
let secondaryCondition: any = "''";

if (
[
UITypes.CreatedTime,
Expand All @@ -178,7 +183,10 @@ export default async function applyAggregation({
UITypes.Rollup,
UITypes.Links,
UITypes.ID,
].includes(_column.uidt)
].includes(_column.uidt) ||
[FormulaDataTypes.DATE, FormulaDataTypes.NUMERIC].includes(
parsedFormulaType,
)
) {
secondaryCondition = 'NULL';
} else if ([UITypes.Rating].includes(_column.uidt)) {
Expand Down Expand Up @@ -224,7 +232,10 @@ export default async function applyAggregation({
UITypes.Rollup,
UITypes.Links,
UITypes.ID,
].includes(_column.uidt)
].includes(_column.uidt) ||
[FormulaDataTypes.DATE, FormulaDataTypes.NUMERIC].includes(
parsedFormulaType,
)
) {
aggregationSql = knex.raw(
`COUNT(*) FILTER (WHERE (??) IS NOT NULL) AS ??`,
Expand Down Expand Up @@ -266,7 +277,10 @@ export default async function applyAggregation({
UITypes.Rollup,
UITypes.Links,
UITypes.ID,
].includes(_column.uidt)
].includes(_column.uidt) ||
[FormulaDataTypes.DATE, FormulaDataTypes.NUMERIC].includes(
parsedFormulaType,
)
) {
aggregationSql = knex.raw(
`COUNT(DISTINCT (??)) FILTER (WHERE (??) IS NOT NULL) AS ??`,
Expand Down Expand Up @@ -312,7 +326,10 @@ export default async function applyAggregation({
UITypes.Rollup,
UITypes.Links,
UITypes.ID,
].includes(_column.uidt)
].includes(_column.uidt) ||
[FormulaDataTypes.DATE, FormulaDataTypes.NUMERIC].includes(
parsedFormulaType,
)
) {
aggregationSql = knex.raw(
`(COUNT(*) FILTER (WHERE (??) IS NOT NULL) * 100.0 / NULLIF(COUNT(*), 0)) AS ??`,
Expand Down Expand Up @@ -352,7 +369,10 @@ export default async function applyAggregation({
UITypes.Rollup,
UITypes.Links,
UITypes.ID,
].includes(_column.uidt)
].includes(_column.uidt) ||
[FormulaDataTypes.DATE, FormulaDataTypes.NUMERIC].includes(
parsedFormulaType,
)
) {
aggregationSql = knex.raw(
`(COUNT(DISTINCT (??)) FILTER (WHERE (??) IS NOT NULL) * 100.0 / NULLIF(COUNT(*), 0)) AS ??`,
Expand Down