sql: v22.2.0: transaction committed but schema change aborted with error: node scan with MaxCost added to the memo

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3894036667/?referrer=webhooks_plugin

Panic message:

coster.go:565: transaction committed but schema change aborted with error: (×): ×: node scan with MaxCost added to the memo
(1) issue: #42061
Wraps: (2) ×
×
Wraps: (3) candidate pg code: XXA00
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:3012
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:3033
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2103
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (5) transaction committed but schema change aborted with error: (×)
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:892
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:471
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:522
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:632
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:684
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:670
| github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.CountLeases
| github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/count.go:43
| github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).WaitForOneVersion
| github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:154
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func2
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:140
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:176
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:49
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).done
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1348
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1904
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:782
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2619
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2753
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1242
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1243
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:410
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:330
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (7) ×
Wraps: (8) assertion failure
Wraps: (9)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*coster).ComputeCost
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/coster.go:565
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:583
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:515
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeExpr
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:291
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroupMember
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:570
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).optimizeGroup
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:515
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform.(*Optimizer).Optimize
| github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:261
| github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).reuseMemo
| github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:491
| github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo
| github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:521
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
| github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:231
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1432
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1058
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:687
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2382
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:218
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1998
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2000
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:217
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (10) node scan with MaxCost added to the memo
Error types: (1) *issuelink.withIssueLink (2) *hintdetail.withHint (3) *pgerror.withCandidateCode (4) *withstack.withStack (5) *errutil.withPrefix (6) *errbase.opaqueWrapper (7) *errutil.withPrefix (8) *assert.withAssertionFailure (9) ...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 3011 to 3013 in 77667a1

	// error message, in case it was a serious error.
	newErr := pgerror.Wrapf(err,
	pgcode.TransactionCommittedWithSchemaChangeFailure,

in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2

cockroach/pkg/sql/conn_executor.go

Lines 3032 to 3034 in 77667a1

	); err != nil {
	handleErr(err)
	}

in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper

cockroach/pkg/sql/conn_executor.go

Lines 2102 to 2104 in 77667a1

	var err error
	advInfo, err = ex.txnStateTransitionsApplyWrapper(ev, payload, res, pos)
	if err != nil {

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1845 to 1847 in 77667a1

	var err error
	if err = ex.execCmd(); err != nil {
	if errors.IsAny(err, io.EOF, errDrainingComplete) {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 827 to 829 in 77667a1

	}(ctx, h)
	return h.ex.run(ctx, s.pool, reserved, cancel)
	}

in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in 77667a1

	reservedOwned = false // We're about to pass ownership away.
	retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
	}()

in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2 at line 3012
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper at line 3033
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2103
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594

Tag	Value
Cockroach Release	v22.2.0
Cockroach SHA:	77667a1
Platform	linux amd64
Distribution	CCL
Environment	v22.2.0
Command	start-single-node
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-23693

[ajwerner]

@cockroachdb/sql-queries folks, do y'all know about node scan with MaxCost added to the memo? This is happening in a place we really do not expect to see such an error. Namely:

cockroach/pkg/sql/catalog/lease/count.go

Lines 30 to 46 in 2f81142

	var whereClauses []string
	for _, t := range versions {
	whereClauses = append(whereClauses,
	fmt.Sprintf(`("descID" = %d AND version = %d AND expiration > $1)`,
	t.ID, t.Version),
	)
	}
	stmt := fmt.Sprintf(`SELECT count(1) FROM system.public.lease AS OF SYSTEM TIME '%s' WHERE `,
	at.AsOfSystemTime()) +
	strings.Join(whereClauses, " OR ")
	values, err := executor.QueryRowEx(
	ctx, "count-leases", nil, /* txn */
	sessiondata.RootUserSessionDataOverride,
	stmt, at.GoTime(),
	)

I'm moving this off the schema board for the moment. The fact that it got wrapped in a schema error is sort of irrelevant I think.

[DrewKimball]

The only way I can see that a Scan cost could be infinity (or close enough to trigger the panic) is if the estimated row-count was very large too... I don't see any reason why we should ever estimate that many rows, but AFAIK we don't technically have any guardrails that would prevent it from happening.

[rytaft]

I've seen this before due to dividing by 0 .... usually it's a bug somewhere in the statisticsBuilder or coster.

[rytaft]

Maybe we can improve the error message here to make it easier to understand what's going on. We could possibly include a serialized representation of the partially constructed memo.

[github-actions]

We have marked this issue as stale because it has been inactive for
12 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy.

[rytaft]

I'm closing this as it's from an unsupported version and we don't have a reproduction. If it happens again on a newer version we can investigate.