Gitea crash loop every 2 minutes

[adyanth]

Gitea Version

1.16.2

Git Version

2.30.2

Operating System

k3s

How are you running Gitea?

Running on Kubernetes v1.22.6+k3s1 with the docker.io/gitea/gitea:1.16.2 image

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

REMOVED

Description

Gitea crashes every 1-2 minutes. All crashes end with [F] Unable to create internal queue for XXX Error: Unable to create queue level for XXX with cfg ... in the logs

Screenshots

No response

[lunny]

Could you paste your queue settings in your app.ini?

[adyanth]

I don't have any section called queue. I can see configs for it here, am I supposed to set anything? https://docs.gitea.io/en-us/config-cheat-sheet/#queue-queue-and-queue

[zeripath]

Could you paste the rest of your app.ini?

[adyanth]

Sure, here you go.

RUN_MODE = prod
APP_NAME = Gitea (Adyanth)

[server]
LFS_JWT_SECRET         = REDACTED
APP_DATA_PATH          = /data
SSH_DOMAIN             = git-ssh.adyanth.site
PROTOCOL               = http
HTTP_PORT              = 3000
NABLE_FEDERATED_AVATAR = true
LANDING_PAGE           = explore
ROOT_URL               = https://git.adyanth.site/
SSH_LISTEN_PORT        = 22
LFS_START_SERVER       = true
SSH_PORT               = 22
ENABLE_PPROF           = false
DOMAIN                 = git.adyanth.site

[ui]
DEFAULT_THEME = arc-green

[mailer]
HOST    = REDACTED
USER    = REDACTED
SUBJECT = %(APP_NAME)s
ENABLED = true
PASSWD  = REDACTED
FROM    = REDACTED

[service]
REQUIRE_SIGNIN_VIEW              = false
REGISTER_EMAIL_CONFIRM           = false
NO_REPLY_ADDRESS                 = REDACTED
ENABLE_NOTIFY_MAIL               = false
DISABLE_REGISTRATION             = true
ALLOW_ONLY_EXTERNAL_REGISTRATION = false

[cron.resync_all_sshkeys]
RUN_AT_START = true
ENABLED      = true

[database]
DB_TYPE = postgres
HOST    = gitea-postgresql.gitea.svc.cluster.local:5432
PASSWD  = REDACTED
NAME    = gitea
USER    = gitea

[openid]
ENABLE_OPENID_SIGNUP = false

[cache]
ENABLED = false
HOST    = gitea-memcached.gitea.svc.cluster.local:11211
ADAPTER = memcache

[security]
REVERSE_PROXY_LIMIT           = 3
INSTALL_LOCK                  = true
SECRET_KEY                    = REDACTED
INTERNAL_TOKEN                = REDACTED
REVERSE_PROXY_TRUSTED_PROXIES = 10.42.0.0/16

[log]
LEVEL = debug

[repository]
ROOT = /data/git/gitea-repositories

[time]
DEFAULT_UI_LOCATION = Asia/Kolkata

[metrics]
ENABLED = false

[oauth2]
JWT_SECRET = REDACTED

[zeripath]

Can Gitea write to:

/data/queues/common

[adyanth]

Let me give that a try. It is via a PVC to an NFS share which I recently enabled with no_root_squash, so I will check

[adyanth]

It can, but the user might be different now (I had removed runas git to troubleshoot). Should I run as git again?

❯ k -n gitea exec gitea-0 -it -- sh
Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
/ # cd /data/queues/common/
/data/queues/common # ls
000002.ldb       CURRENT          LOCK             MANIFEST-000014
000013.log       CURRENT.bak      LOG
/data/queues/common # touch test
/data/queues/common # ls -alh
total 20K    
drwxr-xr-x    1 git      git          128 Feb 26 17:07 .
drwxr-xr-x    1 git      git           12 Feb 25 10:24 ..
-rw-r--r--    1 git      git          167 Feb 25 10:27 000002.ldb
-rw-r--r--    1 git      git            0 Feb 25 11:49 000013.log
-rw-r--r--    1 git      git           16 Feb 25 11:49 CURRENT
-rw-r--r--    1 git      git           16 Feb 25 11:49 CURRENT.bak
-rw-r--r--    1 git      git            0 Feb 25 10:24 LOCK
-rw-r--r--    1 git      git         3.4K Feb 25 11:50 LOG
-rw-r--r--    1 git      git           71 Feb 25 11:50 MANIFEST-000014
-rw-rw-rw-    1 root     root           0 Feb 26 17:07 test
/data/queues/common # 

[zeripath]

Are you trying to have multiple giteas run at the same time?

[zeripath]

If so you need to use redis for your queues

[adyanth]

Nope, just one. When I had this issue, I started removing stuff to isolate. I disabled memcache and run as git.

[zeripath]

Killall Giteas and tell me if the LOCK file goes away

[adyanth]

Scaled it to 0, I still see the files, even the LOCK

[zeripath]

OK if you're definitely sure that there are no Giteas running you could try just deleting the LOCK file, and if things still fail delete the /data/queues/common directory.

If however you want scaling you will need to change your queue type to redis.

[adyanth]

Will give that a try. I do not scale to more than 1, and don't plan to. But will keep redis in mind.

[adyanth]

Removed the LOCK, started gitea, the LOCK file is back. And it has been stable for > 5 mins. Looks like that was it!

[adyanth]

I scaled it back to 0, the LOCK file was not deleted. Scaling back to 1 still looks stable at 2+ minutes. Not sure triggered the issue then, weird. Thanks for your help @zeripath @lunny !

[zeripath]

Cool. I think we can close this. Sorry it took me so long to get involved.

The logs were saying that the issue was the level db was temporarily unavailable - which is code for it can't get a LOCK on the db - and it was pointing to the directory but I guess that this isn't completely clear. I guess the question is whether can do more to make things clearer.

There are a couple of reasons why a lock can't be taken - but I guess we could just detect this error and try to suggest a few things.

I'll close this issue though.