Issue running standard Docker Compose instance of bbapps in CLUSTER_MODE #11814

keith-ellis-sillek · 2023-09-15T16:59:53Z

keith-ellis-sillek
Sep 15, 2023

Checklist

[ X] I have searched budibase discussions and github issues to check if my issue already exists

Hosting

Self
- Method: docker compose on digital ocean:
- Budibase Version: lastest, but appears problem as been since 2.9.39
- App Version: N/A

Describe the bug
If I do a standard install of Budibase through the docker-compose method and enable clustering with CLUSTER_MODE=1 and the pm2 file, the bbworker container starts up in cluster mode with no issue. However, bbapps fails to start with an error.

To Reproduce
I have installed Docker and Docker composed as described here: https://github.com/Budibase/budibase/tree/master/hosting/scripts/linux
I have copied the docker-compose.yaml from here: https://raw.githubusercontent.com/Budibase/budibase/master/hosting/docker-compose.yaml
I have copied the .env files from here: https://raw.githubusercontent.com/Budibase/budibase/master/hosting/.env
From here, you can see that all I did was a "pull" and "up -d" and it works as expected.

root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker-compose pull
WARNING: The OFFLINE_MODE variable is not set. Defaulting to a blank string.
Pulling minio-service      ... done
Pulling couchdb-service    ... done
Pulling couch-init         ... done
Pulling redis-service      ... done
Pulling worker-service     ... done
Pulling app-service        ... done
Pulling proxy-service      ... done
Pulling watchtower-service ... done
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker-compose up -d
WARNING: The OFFLINE_MODE variable is not set. Defaulting to a blank string.
Creating network "root_default" with the default driver
Creating root_minio-service_1      ... done
Creating root_watchtower-service_1 ... done
Creating root_couchdb-service_1    ... done
Creating root_redis-service_1      ... done
Creating root_couch-init_1         ... done
Creating bbworker                  ... done
Creating bbapps                    ... done
Creating bbproxy                   ... done
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker logs bbapps
yarn run v1.22.19
$ node dist/index.js
{"level":"INFO","timestamp":"2023-09-15T16:34:44.809Z","service":"@budibase/server","pid":30,"tenantId":"default","msg":"Budibase running on {\"address\":\"::\",\"family\":\"IPv6\",\"port\":4002}"}
{"level":"ERROR","timestamp":"2023-09-15T16:34:44.821Z","service":"@budibase/server","pid":30,"tenantId":"default","msg":"(node:30) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"INFO","timestamp":"2023-09-15T16:34:45.381Z","service":"@budibase/server","pid":30,"tenantId":"default","msg":"Checking migrations"}
{"level":"INFO","timestamp":"2023-09-15T16:34:45.728Z","service":"@budibase/server","pid":30,"tenantId":"default","msg":"Migrations complete"}
{"level":"ERROR","timestamp":"2023-09-15T16:34:45.744Z","service":"@budibase/server","pid":30,"tenantId":"default","msg":"Failed server startup - Cannot read properties of undefined (reading 'init')"}
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker logs bbworker
yarn run v1.22.19
$ node dist/index.js
{"level":"INFO","timestamp":"2023-09-15T16:34:42.884Z","service":"@budibase/worker","pid":29,"tenantId":"default","msg":"Worker running on {\"address\":\"::\",\"family\":\"IPv6\",\"port\":4003}"}

I then do a docker compose "down", edit my compose yaml to include CLUSTER_MODE. In addition, I add a pm2 file.

services:
  app-service:
    restart: unless-stopped
    image: budibase.docker.scarf.sh/budibase/apps
    container_name: bbapps
    environment:
      CLUSTER_MODE: 1
      SELF_HOSTED: 1
      COUCH_DB_URL: http://${COUCH_DB_USER}:${COUCH_DB_PASSWORD}@couchdb-service:5984
      WORKER_URL: http://worker-service:4003
      MINIO_URL: http://minio-service:9000
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}
      BUDIBASE_ENVIRONMENT: ${BUDIBASE_ENVIRONMENT}
      PORT: 4002
      API_ENCRYPTION_KEY: ${API_ENCRYPTION_KEY}
      JWT_SECRET: ${JWT_SECRET}
      LOG_LEVEL: info
      SENTRY_DSN: https://a34ae347621946bf8acded18e5b7d4b8@o420233.ingest.sentry.io/5338131
      ENABLE_ANALYTICS: "true"
      REDIS_URL: redis-service:6379
      REDIS_PASSWORD: ${REDIS_PASSWORD}
      BB_ADMIN_USER_EMAIL: ${BB_ADMIN_USER_EMAIL}
      BB_ADMIN_USER_PASSWORD: ${BB_ADMIN_USER_PASSWORD}
      PLUGINS_DIR: ${PLUGINS_DIR}
      OFFLINE_MODE: ${OFFLINE_MODE}
    depends_on:
      - worker-service
      - redis-service
    volumes:
      - ./pm2.config.js:/app/pm2.config.js
#      - /some/path/to/plugins:/plugins

  worker-service:
    restart: unless-stopped
    image: budibase.docker.scarf.sh/budibase/worker
    container_name: bbworker
    environment:
      CLUSTER_MODE: 1
      SELF_HOSTED: 1
      PORT: 4003
      CLUSTER_PORT: ${MAIN_PORT}
      API_ENCRYPTION_KEY: ${API_ENCRYPTION_KEY}
      JWT_SECRET: ${JWT_SECRET}
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      MINIO_URL: http://minio-service:9000
      APPS_URL: http://app-service:4002
      COUCH_DB_USERNAME: ${COUCH_DB_USER}
      COUCH_DB_PASSWORD: ${COUCH_DB_PASSWORD}
      COUCH_DB_URL: http://${COUCH_DB_USER}:${COUCH_DB_PASSWORD}@couchdb-service:5984
      SENTRY_DSN: https://a34ae347621946bf8acded18e5b7d4b8@o420233.ingest.sentry.io/5338131
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}
      REDIS_URL: redis-service:6379
      REDIS_PASSWORD: ${REDIS_PASSWORD}
      OFFLINE_MODE: ${OFFLINE_MODE}
    depends_on:
      - redis-service
      - minio-service
      - couch-init
    volumes:
      - ./pm2.config.js:/app/pm2.config.js

My pm2 file looks like this:

module.exports = {
  apps: [
    {
      script: "./dist/index.js",
      instances: "max",
      exec_mode: "cluster",
    },
  ],
}

DOCKER-COMPOSE UP AGAIN - Look at the logs and you see for the bbapps container, it's starting on "null" and has an error. bbworker starts as expected.

root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker-compose up -d
WARNING: The OFFLINE_MODE variable is not set. Defaulting to a blank string.
Creating network "root_default" with the default driver
Creating root_redis-service_1      ... done
Creating root_minio-service_1      ... done
Creating root_watchtower-service_1 ... done
Creating root_couchdb-service_1    ... done
Creating root_couch-init_1         ... done
Creating bbworker                  ... done
Creating bbapps                    ... done
Creating bbproxy                   ... done
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker logs bbapps
yarn run v1.22.19
$ pm2-runtime start pm2.config.js
2023-09-15T16:43:00: PM2 log: Launching in no daemon mode
2023-09-15T16:43:00: PM2 log: App [index:0] starting in -cluster mode-
2023-09-15T16:43:00: PM2 log: App [index:0] online
2023-09-15T16:43:00: PM2 log: App [index:1] starting in -cluster mode-
2023-09-15T16:43:00: PM2 log: App [index:1] online
{"level":"INFO","timestamp":"2023-09-15T16:43:10.300Z","service":"@budibase/server","pid":48,"tenantId":"default","msg":"Budibase running on null"}
{"level":"INFO","timestamp":"2023-09-15T16:43:10.303Z","service":"@budibase/server","pid":41,"tenantId":"default","msg":"Budibase running on null"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:10.312Z","service":"@budibase/server","pid":41,"tenantId":"default","msg":"Failed server startup - Cannot read properties of null (reading 'port')"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:10.314Z","service":"@budibase/server","pid":48,"tenantId":"default","msg":"Failed server startup - Cannot read properties of null (reading 'port')"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:10.321Z","service":"@budibase/server","pid":41,"tenantId":"default","msg":"(node:41) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:10.324Z","service":"@budibase/server","pid":48,"tenantId":"default","msg":"(node:48) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:14.935Z","service":"@budibase/server","pid":68,"tenantId":"default","msg":"(node:68) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:15.004Z","service":"@budibase/server","pid":67,"tenantId":"default","msg":"(node:67) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:15.285Z","service":"@budibase/server","pid":81,"tenantId":"default","msg":"(node:81) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
{"level":"ERROR","timestamp":"2023-09-15T16:43:15.332Z","service":"@budibase/server","pid":80,"tenantId":"default","msg":"(node:80) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.\n\nPlease migrate your code to use AWS SDK for JavaScript (v3).\nFor more information, check the migration guide at https://a.co/7PzMCcy\n(Use `node --trace-warnings ...` to show where the warning was created)"}
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~# docker logs bbworker
yarn run v1.22.19
$ pm2-runtime start pm2.config.js
2023-09-15T16:42:59: PM2 log: Launching in no daemon mode
2023-09-15T16:43:00: PM2 log: App [index:0] starting in -cluster mode-
2023-09-15T16:43:00: PM2 log: App [index:0] online
2023-09-15T16:43:00: PM2 log: App [index:1] starting in -cluster mode-
2023-09-15T16:43:00: PM2 log: App [index:1] online
{"level":"INFO","timestamp":"2023-09-15T16:43:06.978Z","service":"@budibase/worker","pid":47,"tenantId":"default","msg":"Worker running on {\"address\":\"::\",\"family\":\"IPv6\",\"port\":4003}"}
{"level":"INFO","timestamp":"2023-09-15T16:43:06.985Z","service":"@budibase/worker","pid":40,"tenantId":"default","msg":"Worker running on {\"address\":\"::\",\"family\":\"IPv6\",\"port\":4003}"}
root@ubuntu-s-2vcpu-4gb-120gb-intel-nyc3-01:~#

Expected behavior
Expect bbapps to start similar to bbworker

Screenshots
Log have been included in my "Reproduce" above.

App Export
No apps installed

Desktop (please complete the following information):

OS: Mac
Browser Chrome
Version latest

Additional context
Add any other context about the problem here.

keith-ellis-sillek · 2023-09-24T19:20:20Z

keith-ellis-sillek
Sep 24, 2023
Author

So I did some digging and it appears in startup.ts, it resets the PORT environment variable to whatever http server is producting:

export async function startup(app?: any, server?: any) { if (STARTUP_RAN) { return } STARTUP_RAN = true if (server) { //HERE HERE HERE HERE //console.log(Budibase running on ${JSON.stringify(server.address())}) //env._set("PORT", server.address().port) } console.log("env.port" + env.PORT) eventEmitter.emitPort(env.PORT) fileSystem.init() await redis.init() eventInit() initialiseWebsockets(app, server)

I think in the past version, it started listener and waited until complete before it got here. I'm guessing because of pm2 spinning up new processes. However, the new version doesn't seem to wait. Interesting still, if I just comment out the setting of the PORT, it appears to work fine. I notice multiple processes starting and can see them randomly being hit when hitting the server.

Is this something they can do without; if not for the single node instance, at least for the pm2-initiated version? If it will work for the latter, could a if statement be added to ignore if "CLUSTER_MODE" is true?

0 replies

keith-ellis-sillek · 2023-10-02T23:31:26Z

keith-ellis-sillek
Oct 2, 2023
Author

Any word on this?

0 replies

StarZeus · 2023-10-06T14:46:13Z

StarZeus
Oct 6, 2023

I deploy budibase in my k8s environment. And the app-service pod is failing because of this. Any update on this? does anyone has any work around?

3 replies

shogunpurple Oct 6, 2023
Maintainer

@StarZeus are you using CLUSTER_MODE in k8s? I would turn that off, as due to running in a K8S environment you don't need to be running CLUSTER_MODE on your pods

StarZeus Oct 6, 2023

I am not trying to run in CLUSTER_MODE. If I don't specify CLUSTER_MODE in my env var. Does it default it to "true"? Can you tell me how to explicitly turn CLUSTER_MODE off?

shogunpurple Oct 7, 2023
Maintainer

Nope, it should default to off, so it's odd that you would see pm2 errors in your logs if it's not on

keith-ellis-sillek · 2023-10-06T21:21:24Z

keith-ellis-sillek
Oct 6, 2023
Author

This is true. You would want to set you vcpu limits on your budibase containers to 1 and just spin up more containers... if I'm correct. However, for anyone running a straight docker container on a multi core processor, this is going to cause problems.

4 replies

StarZeus Oct 6, 2023

Yes I have set the vcpu limit and setup autoscaling.

keith-ellis-sillek Oct 9, 2023
Author

Oh... it's doing it without cluster mode... strange

StarZeus Oct 9, 2023

Yes. And I had to revert budibase version back to 2.9.38. I will not be able to upgrade to the recent version until the issue is resolved.

keith-ellis-sillek Oct 9, 2023
Author

That's what I had to do... I've thought about going and submitting a "fix" where it skips the "console.out" that I mention above when CLUSTER_MODE is set to true. See if the developers would accept it?!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue running standard Docker Compose instance of bbapps in CLUSTER_MODE #11814

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issue running standard Docker Compose instance of bbapps in CLUSTER_MODE #11814

keith-ellis-sillek Sep 15, 2023

Replies: 4 comments · 7 replies

keith-ellis-sillek Sep 24, 2023 Author

keith-ellis-sillek Oct 2, 2023 Author

StarZeus Oct 6, 2023

shogunpurple Oct 6, 2023 Maintainer

StarZeus Oct 6, 2023

shogunpurple Oct 7, 2023 Maintainer

keith-ellis-sillek Oct 6, 2023 Author

StarZeus Oct 6, 2023

keith-ellis-sillek Oct 9, 2023 Author

StarZeus Oct 9, 2023

keith-ellis-sillek Oct 9, 2023 Author

keith-ellis-sillek
Sep 15, 2023

Replies: 4 comments 7 replies

keith-ellis-sillek
Sep 24, 2023
Author

keith-ellis-sillek
Oct 2, 2023
Author

StarZeus
Oct 6, 2023

shogunpurple Oct 6, 2023
Maintainer

shogunpurple Oct 7, 2023
Maintainer

keith-ellis-sillek
Oct 6, 2023
Author

keith-ellis-sillek Oct 9, 2023
Author

keith-ellis-sillek Oct 9, 2023
Author