Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nakama server cannot apply migrations with beta-20170420 #15269

Closed
novabyte opened this issue Apr 22, 2017 · 7 comments · Fixed by #15511
Closed

Nakama server cannot apply migrations with beta-20170420 #15269

novabyte opened this issue Apr 22, 2017 · 7 comments · Fixed by #15511
Assignees
Labels
O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs
Milestone

Comments

@novabyte
Copy link

I cannot run a migration with Nakama which worked with beta-20170330.

The start of the log file for cockroachdb:

I170422 15:47:49.075609 1 util/log/clog.go:990  [config] file created at: 2017/04/22 15:47:49
I170422 15:47:49.075609 1 util/log/clog.go:990  [config] running on machine: localhost
I170422 15:47:49.075609 1 util/log/clog.go:990  [config] binary: CockroachDB CCL beta-20170420 (darwin amd64, built 2017/04/21 07:53:42, go1.8.1)
I170422 15:47:49.075609 1 util/log/clog.go:990  [config] arguments: [cockroach start --insecure --store=attrs=ssd,path=/tmp/cockroach]

The error output for Nakama server:

{"level":"info","ts":"2017-04-22T15:54:19.704+0100","msg":"Database connection","dsns":"root@localhost:26257"}
{"level":"info","ts":"2017-04-22T15:54:19.713+0100","msg":"Using existing database","name":"nakama"}
{"level":"fatal","ts":"2017-04-22T15:54:19.720+0100","msg":"Failed to apply migrations","count":0,"error":"pq: there is already a transaction in progress handling 20170115200001_initial_schema.sql","stacktrace":"nakama/vendor/go.uber.org/zap.Stack\n\tvendor/go.uber.org/zap/field.go:209\nnakama/vendor/go.uber.org/zap.(*Logger).check\n\tvendor/go.uber.org/zap/logger.go:273\nnakama/vendor/go.uber.org/zap.(*Logger).Fatal\n\tvendor/go.uber.org/zap/logger.go:208\nnakama/cmd.(*migrationService).up\n\tcmd/migrate.go:166\nnakama/cmd.(*migrationService).(nakama/cmd.up)-fm\n\tcmd/migrate.go:94\nnakama/cmd.MigrateParse\n\tcmd/migrate.go:155\nmain.main\n\tmain.go:65"}

The important error message is "pq: there is already a transaction in progress handling 20170115200001_initial_schema.sql". I'm not sure what we'd need to change to apply our schema migrations with the latest version of cockroachdb. The SQL migration we try to apply is here:

https://github.com/heroiclabs/nakama/blob/master/migrations/20170115200001_initial_schema.sql

Let me know what more information I can supply to help identify root cause and how best to resolve it.

@petermattis petermattis added this to the 1.0 milestone Apr 22, 2017
@dianasaur323 dianasaur323 added high priority O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs and removed high priority labels Apr 22, 2017
@vivekmenezes
Copy link
Contributor

It's hard for me to see what particular schema change in that file is failing within your golang code. I'd appreciate it if you can provide me with a pointer your golang code that is failing while applying the schema. This is the code that reads that schema file and calls into the sql driver. Thanks!

@petermattis
Copy link
Collaborator

Can you cat schema.sql | ./cockroach sql?

@vivekmenezes
Copy link
Contributor

vivekmenezes commented Apr 27, 2017

thanks @petermattis

adding

CREATE DATABASE nakama;

SET DATABASE=nakama;

to the initial_schema.sql file and running

$ cat nakama.sql | ./cockroach sql --insecure
CREATE DATABASE
SET
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TABLE
CREATE INDEX
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE

I'm not seeing an issue here. Thanks

@novabyte
Copy link
Author

@vivekmenezes We use this library to do migrations: https://github.com/rubenv/sql-migrate

It depends on gorp (unfortunately) right now. The code we use to drive the migrations is within one of our subcommands of the server:

https://github.com/heroiclabs/nakama/blob/master/cmd/migrate.go#L159

@vivekmenezes
Copy link
Contributor

this issue is caused by the new code we put in to prevent statements from following a schema change in the same transaction when the schema change needs to be rolled out. I'm thinking about how we could fix this. One last minute option is to rollback #14368

@tamird
Copy link
Contributor

tamird commented Apr 27, 2017

@vivekmenezes
Copy link
Contributor

vivekmenezes commented Apr 28, 2017 via email

vivekmenezes added a commit to vivekmenezes/cockroach that referenced this issue Apr 29, 2017
rolls back cockroachdb#15339 cockroachdb#14368 for the most part and cockroachdb#14619 partially.

We used a very big hammer to prevent some statements from
following a schema change. This was done to reduce surprises
but it ended up introducing more surprises. ORMs and schema
migration tools depend on multiple schema changes being
staged in the same transaction.

fixes cockroachdb#15269
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants