Nexcloud down and daily backup stuck

[marcello-dev]

Steps to reproduce

1. Wait for the daily backup to complete

Expected behavior

The backup should start at 4am UTC and finish after a few minutes

Actual behavior

The backup is still ongoing at 2pm UTC (~10hours later)

Host OS

Ubuntu 22.04

Nextcloud AIO version

v8.2.1

Current channel

Can't see it, there is only this message:

Other valuable info

Not sure if relevant but the AIO data dir is on an external ssd, and the backup is stored in a second external drive. I checked the drives and they are fine. I also don't see rsync running, I don't know why the dashboard says backup in progress.

Master container logs:

Daily backup script has started
grep: write error: Broken pipe
Connection to nextcloud-aio-apache (172.18.0.10) 11000 port [tcp/*] succeeded!
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Could not get digest of container nextcloud/aio-watchtower:latest cURL error 28: SSL connection timeout (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://registry-1.docker.io/v2/nextcloud/aio-watchtower/manifests/latest
Not pulling the image for the nextcloud/aio-watchtower container because docker hub does not seem to be reachable.
Waiting for watchtower to stop
Waiting for watchtower to stop
Creating daily backup...
Deleting duplicate sessions
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException

4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.

Watch tower logs:

time="2024-05-30T13:47:43Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-05-30T13:47:44Z" level=debug msg="Making sure everything is sane before starting"
time="2024-05-30T13:47:44Z" level=info msg="Watchtower 1.7.1"
time="2024-05-30T13:47:44Z" level=info msg="Using no notifications"
time="2024-05-30T13:47:44Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2024-05-30T13:47:44Z" level=info msg="Running a one time update."
time="2024-05-30T13:47:44Z" level=debug msg="Checking containers for updated images"
time="2024-05-30T13:47:44Z" level=debug msg="Retrieving running containers"
time="2024-05-30T13:47:44Z" level=debug msg="Trying to load authentication credentials." container=/nextcloud-aio-mastercontainer image="nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2024-05-30T13:47:44Z" level=debug msg="Got image name: nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="Checking if pull is needed" container=/nextcloud-aio-mastercontainer image="nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2024-05-30T13:47:45Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2024-05-30T13:47:45Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2024-05-30T13:47:45Z" level=debug msg="Setting scope for auth token" image=docker.io/nextcloud/all-in-one scope="repository:nextcloud/all-in-one:pull"
time="2024-05-30T13:47:45Z" level=debug msg="No credentials found."
time="2024-05-30T13:47:46Z" level=debug msg="Parsing image ref" host=index.docker.io image=nextcloud/all-in-one normalized=docker.io/nextcloud/all-in-one tag=latest
time="2024-05-30T13:47:46Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/nextcloud/all-in-one/manifests/latest"
time="2024-05-30T13:47:46Z" level=debug msg="Found a remote digest to compare with" remote="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc"
time="2024-05-30T13:47:46Z" level=debug msg=Comparing local="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc" remote="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc"
time="2024-05-30T13:47:46Z" level=debug msg="Found a match"
time="2024-05-30T13:47:46Z" level=debug msg="No pull needed. Skipping image."
time="2024-05-30T13:47:46Z" level=debug msg="No new images found for /nextcloud-aio-mastercontainer"
time="2024-05-30T13:47:46Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2024-05-30T13:47:46Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

[marcello-dev]

As @szaimen suggested, removing the file daily_backup_running in the mastercontainer volume inside the data folder allows to cancel the backup. Then reloading AIO dashboard should show the controls again.
After that I restarted the containers and Nextcloud came back online.
The actual backup only took 1 minute to complete, but for some reason something got stuck and Nextcloud stayed offline for hours.
This issue can be closed if you want.

[szaimen]

Thanks for the report! My idea is to add a warning to the AIO interface what to do if it should be stuck.

[jshore1296]

My backup got stuck last night as well. First time this has happened in a few months.

[mrskizzex]

I also encountered this. New installation, first backup went without a problem, secont time it got stuck. I manually killed borg container, deleted whole borg repo and retried from the start with initial backup and after it finished successfully I tried changing 1 file in nextcloud and tried creating new backup manually and it's stuck unfortunately.

Borg container:

Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_085132-nextcloud-aio"

I see that the borg is doing "something" but it's not doing much, during initial backup it was using much more IO around 60Mb read/writes. My backup is 1.2tb so with 400kb it will take approximately forever to scan it at this rate.

EDIT:
Okay so the backup finished but it's extremely slow. I don't think 1h15m backup every day to backup 100mb of changed files is warranted.

Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_085132-nextcloud-aio"
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240604_085132-nextcloud-aio
Archive fingerprint: 6427d553db9d62893e4064693cae2577187ab908421742fd1f2d6dcf648ceb3b
Time (start): Tue, 2024-06-04 08:51:33
Time (end):   Tue, 2024-06-04 10:07:01
Duration: 1 hours 15 minutes 28.08 seconds
Number of files: 699133
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                1.20 TB              1.17 TB            103.17 MB
All archives:                2.39 TB              2.33 TB              1.13 TB

                       Unique chunks         Total chunks
Chunk index:                 1087923              2278692
------------------------------------------------------------------------------
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4
Pruning the archives...
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:                2.39 TB              2.33 TB              1.13 TB

                       Unique chunks         Total chunks
Chunk index:                 1087923              2278692
------------------------------------------------------------------------------
Compacting the archives...
Backup finished successfully on 04.06.2024 - 10:07:10 (01 hours 15 minutes 38 seconds).

Is there a way to disable backup of data directory in this case? It's extremely slow compared to ZFS snapshots.

EDIT2:

Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_210138-nextcloud-aio"
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240604_210138-nextcloud-aio
Archive fingerprint: 1772546d0652f93a3367eeb37d7340321ba19a457cba38ebf9b2b054a44e512c
Time (start): Tue, 2024-06-04 21:01:39
Time (end):   Tue, 2024-06-04 22:17:55
Duration: 1 hours 16 minutes 15.25 seconds
Number of files: 699203
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                1.20 TB              1.17 TB             98.04 MB
All archives:                3.59 TB              3.50 TB              1.13 TB

                       Unique chunks         Total chunks
Chunk index:                 1088473              3416416
------------------------------------------------------------------------------
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4
Pruning the archives...
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:                3.59 TB              3.50 TB              1.13 TB

                       Unique chunks         Total chunks
Chunk index:                 1088473              3416416
------------------------------------------------------------------------------
Compacting the archives...
Backup finished successfully on 04.06.2024 - 22:18:03 (01 hours 16 minutes 25 seconds).

Another 100mb backup took an 1h15m. Is there something wrong here? Shouldn't it be faster?

The system wasnt under any significant load nor it isn't that slow:

Gigabyte Z690 UD
Core i3-12100
Kingston Fury Beast, DDR5, 32 GB, 5200MHz
4x20tb HDD

[DonkeeeyKong]

I am not sure if this is a separate issue or belongs here:

My backup isn't stuck, but my AIO interface shows that it's running even though it has finished. I only noticed this after a reboot when I saw that the nextcloud containers weren't running. Restarting the mastercontainer or the backup container didn't help (the backup ran through, but the containers didn't start). The AIO interface still shows the backup is running. I manually started the nextcloud containers and it's working fine apart from the AIO interface:

Mastercontainer log:

Trying to fix docker.sock permissions internally...
Creating docker group internally with id 988
�[0;92mInitial startup of Nextcloud All-in-One complete!
You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
E.g. https://internal.ip.of.this.server:8080
⚠️ Important: do always use an ip-address if you access this port and not a domain as HSTS might block access to it later!

If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
https://your-domain-that-points-to-this-server.tld:8443�[0m
{"level":"info","ts":1717676142.3378007,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
[06-Jun-2024 12:15:42] NOTICE: fpm is running, pid 161
[06-Jun-2024 12:15:42] NOTICE: ready to handle connections
[Thu Jun 06 12:15:42.345958 2024] [mpm_event:notice] [pid 155:tid 140267496704776] AH00489: Apache/2.4.59 (Unix) OpenSSL/3.1.5 configured -- resuming normal operations
[Thu Jun 06 12:15:42.346007 2024] [core:notice] [pid 155:tid 140267496704776] AH00094: Command line: 'httpd -D FOREGROUND'
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException

4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException

4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.

I am using openmediavault 7 / Debian Bookworm. There was a Docker/containerd update yesterday. Could this be related in any way?

Edit: Removing the file daily_backup_running did the trick for me as well. The next backup didn't get stuck and I got an update notification and succesfully updated Nextcloud from 28.0.5.1 to 28.0.6.1. I still have no idea what caused this problem though...

[szaimen]

Thanks for the report! My idea is to add a warning to the AIO interface what to do if it should be stuck.

This is now released with v9.0.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel

[marcello-dev]

Hello, the backup process is stuck again.
My AIO was automatically upgraded to 8.3.0, and the issue looks a bit different:

1. The Borg container is still running after 3 hours, when it usually takes only a few minutes.
2. Logs shows that it is waiting for the Borg container to stop (see logs below)

Logs:

Daily backup script has started
grep: write error: Broken pipe
Connection to nextcloud-aio-apache (172.18.0.10) 11000 port [tcp/*] succeeded!
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Waiting for watchtower to stop
Creating daily backup...
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
.....
Waiting for backup container to stop
Waiting for backup container to stop
Deleting duplicate sessions
Waiting for backup container to stop
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException

4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
...

I still don't have the feature to manually unblock the backup from the dashboard, because it was implemented for AIO v9. Maybe it is something to investigate?
Also how to know if Borg is still doing something? I did upload some big files yesterday but it should not take that long.

New feature idea: it would be good to add a timestamp next to the log message for better troubleshooting.