Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: coster.go:565: transaction committed but schema change aborted with error: (×): failed to update job status: job 833914667411111937: ×: node scan with MaxCost added to the memo (1) issue: https://gith... #95731

Closed
cockroach-teamcity opened this issue Jan 24, 2023 · 1 comment
Closed

sentry: coster.go:565: transaction committed but schema change aborted with error: (×): failed to update job status: job 833914667411111937: ×: node scan with MaxCost added to the memo (1) issue: https://gith... #95731

cockroach-teamcity opened this issue Jan 24, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 24, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3894046525/?referrer=webhooks_plugin

Panic message:

coster.go:565: transaction committed but schema change aborted with error: (×): failed to update job status: job 833914667411111937: ×: node scan with MaxCost added to the memo
(1) issue: #42061
Wraps: (2) ×
×
Wraps: (3) candidate pg code: XXA00
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:3012
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:3033
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2103
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (5) transaction committed but schema change aborted with error: (×)
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAfterTempIndexMerge.func1
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2219
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func1
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:52
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func3
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:163
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:975
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:961
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:974
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:937
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:916
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:155
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:49
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txn
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2464
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAfterTempIndexMerge
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2175
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).backfillIndexes
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2133
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:361
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1898
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:782
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2619
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2753
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1242
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1243
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:410
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
| github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:330
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (7) failed to update job status
Wraps: (8)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/jobs.(*Job).update
| github.com/cockroachdb/cockroach/pkg/jobs/update.go:296
| github.com/cockroachdb/cockroach/pkg/jobs.(*Job).Update
| github.com/cockroachdb/cockroach/pkg/jobs/update.go:140
| github.com/cockroachdb/cockroach/pkg/jobs.(*Job).RunningStatus
| github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:309
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAfterTempIndexMerge.func1
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2214
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn.func1
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:52
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor.func3
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:163
| github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
| github.com/cockroachdb/cockroach/pkg/kv/db.go:975
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:961
| github.com/cockroachdb/cockroach/pkg/kv.runTxn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:974
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
| github.com/cockroachdb/cockroach/pkg/kv/db.go:937
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
| github.com/cockroachdb/cockroach/pkg/kv/db.go:916
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).TxnWithExecutor
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:155
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*CollectionFactory).Txn
| github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/txn.go:49
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).txn
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2464
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAfterTempIndexMerge
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2175
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).backfillIndexes
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:2133
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill
| github.com/cockroachdb/cockroach/pkg/sql/backfill.go:361
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1898
| github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:782
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2619
| github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
| github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2753
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1242
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1243
|...

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 3011 to 3013 in 77667a1

// error message, in case it was a serious error.
newErr := pgerror.Wrapf(err,
pgcode.TransactionCommittedWithSchemaChangeFailure,
in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2

cockroach/pkg/sql/conn_executor.go

Lines 3032 to 3034 in 77667a1

); err != nil {
handleErr(err)
}
in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper

cockroach/pkg/sql/conn_executor.go

Lines 2102 to 2104 in 77667a1

var err error
advInfo, err = ex.txnStateTransitionsApplyWrapper(ev, payload, res, pos)
if err != nil {
in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1845 to 1847 in 77667a1

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 827 to 829 in 77667a1

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}
in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in 77667a1

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()
in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper.func2 at line 3012
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper at line 3033
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2103
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
Tag Value
Cockroach Release v22.2.0
Cockroach SHA: 77667a1
Platform linux amd64
Distribution CCL
Environment v22.2.0
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-23696
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jan 24, 2023
@yuzefovich
Copy link
Member

dup of #95728

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: coster.go:565: transaction committed but schema change aborted with error: (×): failed to update job status: job 833914667411111937: ×: node scan with MaxCost added to the memo (1) issue: https://gith... sentry: coster.go:565: transaction committed but schema change aborted with error: (×): failed to update job status: job 833914667411111937: ×: node scan with MaxCost added to the memo (1) issue: https://gith... Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity