Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update to 1.50.1 ended in 503 Service Unavailable #1598

Closed
saschabrockel opened this issue Sep 28, 2022 · 24 comments
Closed

Update to 1.50.1 ended in 503 Service Unavailable #1598

saschabrockel opened this issue Sep 28, 2022 · 24 comments
Assignees
@theCalcaholic
Labels
bug disruption Issues actively causing problems with existing installations docker has-updates

Comments

@saschabrockel
Copy link

saschabrockel commented Sep 28, 2022 •
edited
Loading

Watchtower automatically updated the image in the night and as soon as the update was over I got multiple notifications that NCP is down with error code 503. When I go on the website I only see the following:
image

It looks like there is something wrong with the database or its logs.
Edit: there are no log files anymore in the database folder.

System information

NextCloudPi diagnostics 

NextCloudPi version  v1.50.1
NextCloudPi image    NextCloudPi_docker_09-27-22
OS                   Debian GNU/Linux 11. 4.9.312-125 (aarch64)
automount            no
USB devices          sda 
datadir              /data/nextcloud/data
data in SD           no
data filesystem      ext2/ext3
data disk usage      622G/3.6T
rootfs usage         25G/58G
swapfile             /swapfile
dbdir                /data-ro/database
Nextcloud check      ok
Nextcloud version    24.0.5.1
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          down
Postfix service      up
Internet check       ok
public IP            ***REMOVED SENSITIVE VALUE***
public IP            ***REMOVED SENSITIVE VALUE***
Port check 80        open (ipv4)
Port check 443       open (ipv4)
IP                   ***REMOVED SENSITIVE VALUE***
gateway              ***REMOVED SENSITIVE VALUE***
Interface            eth0
certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
Uptime               39days
Nextcloud configuration 

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "5": "nextcloudpi.local",
            "7": "nextcloudpi",
            "8": "nextcloudpi.lan",
            "1": "172.18.0.2",
            "20": ***REMOVED SENSITIVE VALUE***,
            "21": ***REMOVED SENSITIVE VALUE***,
            "22": "_",
            "3": "6ad708428399",
            "11": ***REMOVED SENSITIVE VALUE***,
            "14": "6ad708428399",
            "": ***REMOVED SENSITIVE VALUE***
        },
        "default_language": "de",
        "default_phone_region": "DE",
        "default_locale": "de_DE",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.5.1",
        "overwritehost": ***REMOVED SENSITIVE VALUE***,
        "overwrite.cli.url": "https:\/\/6ad708428399\/",
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/var\/www\/nextcloud\/data\/tmp",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "jpeg_quality": "60",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "maintenance": false,
        "ldapUserCleanupInterval": 720,
        "loglevel": 1,
        "htaccess.RewriteBase": "\/",
        "mail_sendmailmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": "",
        "app_install_overwrite": [
            "keeweb",
            "drop_account",
            "ldap_write_support",
            "files_texteditor"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory"
    }
}
HTTPd logs 

[Wed Sep 28 04:01:51.254465 2022] [ssl:error] [pid 116:tid 547536105488] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=localhost / issuer: CN=localhost / serial: 6BA1FB069D24F2D14A67E58F3801D8065EF0755D / notbefore: Sep 27 21:32:38 2022 GMT / notafter: Sep 24 21:32:38 2032 GMT]
[Wed Sep 28 04:01:51.254502 2022] [ssl:error] [pid 116:tid 547536105488] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Sep 28 04:01:51.271459 2022] [ssl:error] [pid 117:tid 547536105488] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=localhost / issuer: CN=localhost / serial: 6BA1FB069D24F2D14A67E58F3801D8065EF0755D / notbefore: Sep 27 21:32:38 2022 GMT / notafter: Sep 24 21:32:38 2032 GMT]
[Wed Sep 28 04:01:51.271485 2022] [ssl:error] [pid 117:tid 547536105488] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Sep 28 04:01:51.274829 2022] [mpm_event:notice] [pid 117:tid 547536105488] AH00489: Apache/2.4.54 (Debian) OpenSSL/1.1.1n configured -- resuming normal operations
[Wed Sep 28 04:01:51.274898 2022] [core:notice] [pid 117:tid 547536105488] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 08:43:42.295362 2022] [mpm_event:notice] [pid 117:tid 547536105488] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Sep 28 08:43:47.540474 2022] [ssl:error] [pid 118:tid 547657625616] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=localhost / issuer: CN=localhost / serial: 6BA1FB069D24F2D14A67E58F3801D8065EF0755D / notbefore: Sep 27 21:32:38 2022 GMT / notafter: Sep 24 21:32:38 2032 GMT]
[Wed Sep 28 08:43:47.540494 2022] [ssl:error] [pid 118:tid 547657625616] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Sep 28 08:43:47.556524 2022] [ssl:error] [pid 119:tid 547657625616] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=localhost / issuer: CN=localhost / serial: 6BA1FB069D24F2D14A67E58F3801D8065EF0755D / notbefore: Sep 27 21:32:38 2022 GMT / notafter: Sep 24 21:32:38 2032 GMT]
[Wed Sep 28 08:43:47.556568 2022] [ssl:error] [pid 119:tid 547657625616] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Sep 28 08:43:47.559827 2022] [mpm_event:notice] [pid 119:tid 547657625616] AH00489: Apache/2.4.54 (Debian) OpenSSL/1.1.1n configured -- resuming normal operations
[Wed Sep 28 08:43:47.559898 2022] [core:notice] [pid 119:tid 547657625616] AH00094: Command line: '/usr/sbin/apache2'
Database logs 

tail: cannot open '/var/log/mysql/*.log' for reading: No such file or directory
Nextcloud logs 

{"reqId":"Yy6RxfkDpbQB4LUPA2as5AAAAFI","level":1,"time":"2022-09-24T07:12:37+02:00","remoteAddr":"172.18.0.3","user":"--","app":"no app in context","method":"GET","url":"/apps/notes/notes/596824","message":"Unable to generate a URL for the named route \"notes.page.index\" as such route does not exist.","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","version":"24.0.3.2","exception":{"Exception":"Symfony\\Component\\Routing\\Exception\\RouteNotFoundException","Message":"Unable to generate a URL for the named route \"notes.page.index\" as such route does not exist.","Code":0,"Trace":[{"file":"/data/nextcloud/lib/private/Route/Router.php","line":361,"function":"generate","class":"Symfony\\Component\\Routing\\Generator\\UrlGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/Route/CachingRouter.php","line":58,"function":"generate","class":"OC\\Route\\Router","type":"->"},{"file":"/data/nextcloud/lib/private/URLGenerator.php","line":91,"function":"generate","class":"OC\\Route\\CachingRouter","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":287,"function":"linkToRoute","class":"OC\\URLGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":113,"function":"init","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/apps/theming/lib/ThemingDefaults.php","line":199,"function":"getAll","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Defaults.php","line":280,"function":"getShortFooter","class":"OCA\\Theming\\ThemingDefaults","type":"->"},{"file":"/data/nextcloud/lib/public/Defaults.php","line":177,"function":"getLongFooter","class":"OC_Defaults","type":"->"},{"file":"/data/nextcloud/core/templates/layout.guest.php","line":54,"function":"getLongFooter","class":"OCP\\Defaults","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":180,"args":["/data/nextcloud/core/templates/layout.guest.php"],"function":"include"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":150,"function":"load","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":182,"function":"fetchPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":213,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":302,"function":"printPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":973,"function":"checkMaintenanceMode","class":"OC","type":"::"},{"file":"/data/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/data/nextcloud/3rdparty/symfony/routing/Generator/UrlGenerator.php","Line":143,"message":"Unable to generate a URL for the named route \"notes.page.index\" as such route does not exist.","exception":{},"CustomMessage":"Unable to generate a URL for the named route \"notes.page.index\" as such route does not exist."}}
{"reqId":"Yy6RxfkDpbQB4LUPA2as5AAAAFI","level":1,"time":"2022-09-24T07:12:37+02:00","remoteAddr":"172.18.0.3","user":"--","app":"no app in context","method":"GET","url":"/apps/notes/notes/596824","message":"Unable to generate a URL for the named route \"photos.page.index\" as such route does not exist.","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","version":"24.0.3.2","exception":{"Exception":"Symfony\\Component\\Routing\\Exception\\RouteNotFoundException","Message":"Unable to generate a URL for the named route \"photos.page.index\" as such route does not exist.","Code":0,"Trace":[{"file":"/data/nextcloud/lib/private/Route/Router.php","line":361,"function":"generate","class":"Symfony\\Component\\Routing\\Generator\\UrlGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/Route/CachingRouter.php","line":58,"function":"generate","class":"OC\\Route\\Router","type":"->"},{"file":"/data/nextcloud/lib/private/URLGenerator.php","line":91,"function":"generate","class":"OC\\Route\\CachingRouter","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":287,"function":"linkToRoute","class":"OC\\URLGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":113,"function":"init","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/apps/theming/lib/ThemingDefaults.php","line":199,"function":"getAll","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Defaults.php","line":280,"function":"getShortFooter","class":"OCA\\Theming\\ThemingDefaults","type":"->"},{"file":"/data/nextcloud/lib/public/Defaults.php","line":177,"function":"getLongFooter","class":"OC_Defaults","type":"->"},{"file":"/data/nextcloud/core/templates/layout.guest.php","line":54,"function":"getLongFooter","class":"OCP\\Defaults","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":180,"args":["/data/nextcloud/core/templates/layout.guest.php"],"function":"include"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":150,"function":"load","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":182,"function":"fetchPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":213,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":302,"function":"printPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":973,"function":"checkMaintenanceMode","class":"OC","type":"::"},{"file":"/data/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/data/nextcloud/3rdparty/symfony/routing/Generator/UrlGenerator.php","Line":143,"message":"Unable to generate a URL for the named route \"photos.page.index\" as such route does not exist.","exception":{},"CustomMessage":"Unable to generate a URL for the named route \"photos.page.index\" as such route does not exist."}}
{"reqId":"Yy6RxfkDpbQB4LUPA2as5AAAAFI","level":1,"time":"2022-09-24T07:12:37+02:00","remoteAddr":"172.18.0.3","user":"--","app":"no app in context","method":"GET","url":"/apps/notes/notes/596824","message":"Unable to generate a URL for the named route \"tasks.page.index\" as such route does not exist.","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","version":"24.0.3.2","exception":{"Exception":"Symfony\\Component\\Routing\\Exception\\RouteNotFoundException","Message":"Unable to generate a URL for the named route \"tasks.page.index\" as such route does not exist.","Code":0,"Trace":[{"file":"/data/nextcloud/lib/private/Route/Router.php","line":361,"function":"generate","class":"Symfony\\Component\\Routing\\Generator\\UrlGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/Route/CachingRouter.php","line":58,"function":"generate","class":"OC\\Route\\Router","type":"->"},{"file":"/data/nextcloud/lib/private/URLGenerator.php","line":91,"function":"generate","class":"OC\\Route\\CachingRouter","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":287,"function":"linkToRoute","class":"OC\\URLGenerator","type":"->"},{"file":"/data/nextcloud/lib/private/NavigationManager.php","line":113,"function":"init","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/apps/theming/lib/ThemingDefaults.php","line":199,"function":"getAll","class":"OC\\NavigationManager","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Defaults.php","line":280,"function":"getShortFooter","class":"OCA\\Theming\\ThemingDefaults","type":"->"},{"file":"/data/nextcloud/lib/public/Defaults.php","line":177,"function":"getLongFooter","class":"OC_Defaults","type":"->"},{"file":"/data/nextcloud/core/templates/layout.guest.php","line":54,"function":"getLongFooter","class":"OCP\\Defaults","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":180,"args":["/data/nextcloud/core/templates/layout.guest.php"],"function":"include"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":150,"function":"load","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":182,"function":"fetchPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":213,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":302,"function":"printPage","class":"OC\\Template\\Base","type":"->"},{"file":"/data/nextcloud/lib/base.php","line":973,"function":"checkMaintenanceMode","class":"OC","type":"::"},{"file":"/data/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/data/nextcloud/3rdparty/symfony/routing/Generator/UrlGenerator.php","Line":143,"message":"Unable to generate a URL for the named route \"tasks.page.index\" as such route does not exist.","exception":{},"CustomMessage":"Unable to generate a URL for the named route \"tasks.page.index\" as such route does not exist."}}
{"reqId":"vwDqwGTjjlJBXr97608w","level":1,"time":"2022-09-24T07:13:11+02:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::finishedCheckCodeIntegrity: Finished code integrity check","userAgent":"--","version":"24.0.3.2","data":{"app":"updater"}}
{"reqId":"vwDqwGTjjlJBXr97608w","level":1,"time":"2022-09-24T07:13:11+02:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::updateEnd: Update successful","userAgent":"--","version":"24.0.5.1","data":{"app":"updater"}}
{"reqId":"vwDqwGTjjlJBXr97608w","level":1,"time":"2022-09-24T07:13:11+02:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::maintenanceDisabled: Turned off maintenance mode","userAgent":"--","version":"24.0.5.1","data":{"app":"updater"}}
{"reqId":"vwDqwGTjjlJBXr97608w","level":1,"time":"2022-09-24T07:13:11+02:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::resetLogLevel: Reset log level to Info(1)","userAgent":"--","version":"24.0.5.1","data":{"app":"updater"}}
{"reqId":"AaLiFvYqHOvBJwpqewql","level":1,"time":"2022-09-24T07:15:05+02:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Could not find updater directory /data/nextcloud/data/updater-ocpki9ramxdk/backups - cleanup step not needed","userAgent":"--","version":"24.0.5.1","data":[]}
{"reqId":"lp48FI0FIZG7fcnPStaI","level":1,"time":"2022-09-24T21:05:02+02:00","remoteAddr":"","user":"--","app":"calendar","method":"","url":"--","message":"Found and deleted 0 outdated booking confirmations.","userAgent":"--","version":"24.0.5.1","data":{"app":"calendar"}}
{"reqId":"pF3yBFjURzCboOBXnHZ2","level":1,"time":"2022-09-24T21:30:02+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire /Notes/Schuldenliste.txt next version should be 1660611086 or smaller. (prevTimestamp: 1661215886; step: 604800","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"pF3yBFjURzCboOBXnHZ2","level":1,"time":"2022-09-24T21:30:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: /Notes/Schuldenliste.txt.v1660754542","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire /Notes/To-Do.txt next version should be 1660875127 or smaller. (prevTimestamp: 1661479927; step: 604800","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire /Notes/To-Do.txt next version should be 1660875127 or smaller. (prevTimestamp: 1661479927; step: 604800","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Mark to expire /Notes/To-Do.txt next version should be 1660875127 or smaller. (prevTimestamp: 1661479927; step: 604800","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: /Notes/To-Do.txt.v1661310558","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: /Notes/To-Do.txt.v1661220752","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"qY3DwqPkchezO7WrUD2X","level":1,"time":"2022-09-25T05:25:03+02:00","remoteAddr":"","user":"--","app":"files_versions","method":"","url":"--","message":"Expire: /Notes/To-Do.txt.v1661125917","userAgent":"--","version":"24.0.5.1","data":{"app":"files_versions"}}
{"reqId":"Cwj3KQcbHkspuVByr996","level":1,"time":"2022-09-25T21:05:04+02:00","remoteAddr":"","user":"--","app":"calendar","method":"","url":"--","message":"Found and deleted 0 outdated booking confirmations.","userAgent":"--","version":"24.0.5.1","data":{"app":"calendar"}}
{"reqId":"NMG4nRVYT0vjgLFyIFLL","level":1,"time":"2022-09-26T21:10:02+02:00","remoteAddr":"","user":"--","app":"calendar","method":"","url":"--","message":"Found and deleted 0 outdated booking confirmations.","userAgent":"--","version":"24.0.5.1","data":{"app":"calendar"}}
{"reqId":"dFtSODe5ZqUirQwgVzfY","level":1,"time":"2022-09-27T21:15:03+02:00","remoteAddr":"","user":"--","app":"calendar","method":"","url":"--","message":"Found and deleted 0 outdated booking confirmations.","userAgent":"--","version":"24.0.5.1","data":{"app":"calendar"}}
@cb-studi
Copy link

I have exactly the same problem and also informed @theCalcaholic

@saschabrockel
Copy link
Author

@cb-studi
I have seen your issue but I thought it is different because you got database logs and different Redis logs.
For me the Redis logs just look like this:

redis-server is running.

@cb-studi
Copy link

cb-studi commented Sep 28, 2022 •
edited
Loading

@saschabrockel
As soon as I try to access my Nextcloud instance normally with my browser, I get exactly the message you show here. And this is only the case since the update to v.1.50.1. In this respect, both symptoms are probably one single issue. @theChatholic confirmed to me that the redis-server obviously works as it should. It has something to do with the HBP service. I have not made any manual changes to my Nextcloud except for the typical setup and updates. All port configurations are correct as well.

@saschabrockel
Copy link
Author

My NCP instance is old (from Q1 2020 I guess) and the last updates all caused problems for me...

But it is also sad that I just learned today that people who use Docker should never use ncp-update which I have done back then and it also was activated for auto-updating... maybe that has caused some problems over time.

@victor-rays
Copy link
Collaborator

victor-rays commented Sep 28, 2022 •
edited
Loading

Edit: never mind, missed that you guys are using docker

@victor-rays
Copy link
Collaborator

The v1.50.1 image is currently removed, can you guys remove the container & image for v1.50.1 ? 🙏
The current tag i v1.49.1 with NC 24, below is from the image
Nextcloud version 24.0.3.2. NextCloudPi version v1.49.1

@theCalcaholic
Copy link
Collaborator

@saschabrockel could you please follow the steps described in #1593 (comment) and see if that works around your issue as well?

@saschabrockel
Copy link
Author

@theCalcaholic did not work. It does not change anything. But I'm still not sure if we have the same issue. I guess it is something with the database. That was often a pain in the...

@theCalcaholic
Copy link
Collaborator

@saschabrockel Sorry, I forgot to mention: You need to restart the container afterwards

@saschabrockel
Copy link
Author

@theCalcaholic Sorry, and I forgot to mention that I did that of course :D

@theCalcaholic theCalcaholic added bug docker disruption Issues actively causing problems with existing installations and removed has-updates labels Sep 28, 2022
@saschabrockel
Copy link
Author

@theCalcaholic have you recognized my missing MySQL logs?

I had trouble with another MariaDB container for another application and I know it was helpful to add these both as environment variables:

MYSQL_INITDB_SKIP_TZINF=1
MARIADB_AUTO_UPGRADE=1

Is my MySQL database of NC maybe deleted (in parts)?
image

@victor-rays
Copy link
Collaborator

It could be, is it related to #1577 perhaps?

#1577 (comment)

@saschabrockel
Copy link
Author

saschabrockel commented Sep 28, 2022 •
edited
Loading

I am so dumb but I guess this restore completely f*cked me. What kind of restore is this??? All files are gone. Of every user. I would have never thought that this could happen and I have no goddamn copy.

I have used the ncp-backups folder with a backup of the 01.09.2022 with 900MB. While the restore was running I saw that there was a backup of the update from yesterday... but the goddamn restore also deleted this one and the whole folder /data/nextcloud/data/ncp-update-backups.

I feel I'm done with NCP now... how should I ever trust again if a restore deletes all files of all users. How should I explain this to all users now? And the best thing is it is still error 503 service unavailable. I'm so fcking done

@ZendaiOwl couldn't u mention this... #1417
so everything is gone. Wow. No warning about this anywhere.

@theCalcaholic any idea on what happened there and how to get the files back?

@theCalcaholic
Copy link
Collaborator

@saschabrockel Yes, I'm afraid it wasn't the restore that deleted your data, it was the broken update (compare #1577).
Automatic ncp updates should never have been available for docker in the first place (because docker should use a different update mechanism).

I'm not yet entirely sure why the data has been deleted, but I know how it happened for the database. My focus has so far been on preventing this from happening for all the users which didn't update yet. Now I'm investigating ways to recover.

This update was really disastrous and should never have passed my tests

@saschabrockel
Copy link
Author

@theCalcaholic I have not looked into the folders but it looked fine from the outside after the update. Also as you have mentioned the /data/nextcloud/data/ncp-update-backups folder was there before I used the old backup...

I had a backup without data included and I guess I deleted everything with this restore. I am trying to use foremost and multiple recovery tools now to get something back... I thought using a RAID 1 would be enough and would have never thought this could actually happen.

Also weird is that it was said that update 1.50 caused the problem but if I have a look at what Watchtower did there it was updated instantly to 1.50.1 and as I have read that should not have deleted the database.

@victor-rays
Copy link
Collaborator

@saschabrockel I have been mentioning it non-stop .. There are warnings littered over the related topics in the forum and the issues here. I am one person, working full-time and studying full-time, I can't do everything for you

@theCalcaholic
Copy link
Collaborator

@ZendaiOwl couldn't u mention this... #1417

@ZendaiOwl is a volunteer helping with NCP in his free time. That he is as active and knowledgeable as is the case, is amazing. He's not support staff and can't be expected to guide everyone through fixing their NCP instance (nor know any issue that has been brought up within the project).

@saschabrockel
Copy link
Author

saschabrockel commented Sep 28, 2022 •
edited
Loading

Yeah, I understand it. Warnings in forum okay but who would goddamn think that a restore or update can delete all of my files. My whole university career with files like my master thesis was only saved there. You can imagine what this means to me. I have no idea to rescue and I really need to get this back... I have no copy.

I would really really appreciate any help I could get.

@theCalcaholic
Copy link
Collaborator

theCalcaholic commented Sep 29, 2022 •
edited
Loading

@saschabrockel I'm very sorry to hear this. :(
So do I understand correctly that you have 1) no backups other than the nc-update backups and 2) no clients that have the data synchronized to other devices?

@theCalcaholic
Copy link
Collaborator

Btw, I know it's not of much help to you, but I have created an early draft for a fix for ncp-restore (still requires testing, though).

@saschabrockel
Copy link
Author

@theCalcaholic for Nextcloud I don't have any backup except ncp-update-backups. The folder you mentioned in the other post (the automatic update backup folder) was deleted on the restore.

I have luckily some data that is not gone like password files because this would literally cost me everything but my friends had some data there and they were only using it per browser. The same applies to my university stuff of 10 semesters... I mean I have already finished everything but I had tons of information in it and really liked to have them to see things I might forget.

Everything else is not a problem for me. It is really just that one big thing. I deleted from the university OneDrive after I left the university because I did not want that they have all data there... I guess that was a dumb decision.

You can be sure that I'm going to learn now how to back up NextCloud the best way. I have read 2 or 3 years ago about rsync and stuff but I was just too much because I was a bloody beginner. And until today it was no problem.

Yeah but I'm still happy to see the early draft. I wish nobody to have this. Imagine that would happen with a company's data. I know you also do have not this much time and I'm sorry for my rage on some posts before. It was not against @ZendaiOwl. I just could not believe that a ticket of January which I have seen too late cost me this...

I guess I need to set up the whole NextCloud instance from scratch because the restore didn't even work and there is literally nothing left I guess...

I'm trying to use ext4magic now to get some data back but only strange things and little files are found. Tons of .gitkeep, .txt, .PHP and .unknown files. I have the feeling that the files were deleted and overwritten by the new empty folder for each user...

@theCalcaholic
Copy link
Collaborator

theCalcaholic commented Sep 29, 2022 •
edited
Loading

I'm afraid there's only little chance of getting your data back :/
I'm very sorry about that.

I don't mean to excuse the bugs you were encountering, but I can't overstate the importance of a reliable backup strategy. As a starting point, I'd recommend mount another host directory to the container, e.g. -v /opt/ncp-backups:/backups and configure nc-backup-auto to write backups into that directory (which would be /backups/ in the example).

That's only a starting point, because to have really reliable backups, they should at least reside on a separate disk, if not a separate machine/location. But it's a good start :)

@theCalcaholic theCalcaholic self-assigned this Sep 29, 2022
@saschabrockel
Copy link
Author

My problem is or was that I do not have so much free space. I thought having this data on RAID 1 would be enough, to be honest, and I would never imagine that this could happen. And that is the point where I don't know how to have a small or good backup because I can't have 1 TB of data on three separate disks and I guess you understand the point.

But I guess you would say I need at least one drive more for my data saved on RAID 1. The point where you could me a bit is that I don't want to attach a USB hard disk all the time because it is not good for the lifespan of the hard disk. The other disks are NAS disks so it is no problem. But how to backup for example every day with a USB hard disk but only turn it on at this time for this purpose?

And I am not sure yet if I might not get the data back. ext4magic is running now for 20 hours I guess and brought me 32GB back but I don't know exactly what it is yet. Many files seem to be so old I cannot imagine that they were brought back. But the problem is that I have not that much free space on the hard drive the recovered files are now written to and I won't be able to pause the process and say use now another hard drive to save it...

But it seems to be a good tool for Linux users that might have the same case as me now. If anybody reads this here they should give it a try or ask me if they need a short instruction. You can mention my comment here if somebody else will tell you about a similar case.

@theCalcaholic
Copy link
Collaborator

@saschabrockel I wouldn't save on the backup disk, that's too important. Backups are more important than your raid setup so if you already have a raid 1, consider migrating to a single data disk and a single disk for backups.

One option for backups that's very disk space efficient are btrfs snapshots.
However, ncp doesn't support managing them for you if run via docker (yet).
You can mount a btrfs subvolume as data directory though, and manage snapshots yourself.
This can be achieved by

  1. having a btrfs partition or disk mounted somewhere in the host (let's assume /opt/ncp-data)
  2. mounting that directory to your container with -v /opt/data:/data
  3. A cron job or similar that regularly creates snapshots of the data directory

These snapshots can then be copied to a second disk (which is better than raid, because it protects against accidental deletion)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@theCalcaholic theCalcaholic

Labels
bug disruption Issues actively causing problems with existing installations docker has-updates
Projects
📦NextcloudPi
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@theCalcaholic @victor-rays @cb-studi @saschabrockel