Nextcloud 13.0.6: database size exploding, scanning datadir

[linuxrrze]

Steps to reproduce

1. Nextcloud datadir contains other files (in my example: snapshots of nextcloud data in folder .snapshots)
2. Update to nextcloud 13.0.6
3. Wait for cron.php to start

Expected behaviour

Cron.php should "process" files in known users directory and leave .snapshots and containing files alone. Never had trouble with this until 13.0.6

Actual behaviour

Cron.php seems to "process" every single file in this directory.
In my case this means processing all snapshots and containing files.
This made my database size explode from 1 GB to 8 GB (process still running after 40 hours)

Server configuration

Operating system:
Ubuntu 18.04 (x86_64)

Web server:
Apache 2.4.29

Database:
Postgresql 10.5

PHP version:
7.2.7

Nextcloud version: (see Nextcloud admin page)
13.0.6

Updated from an older Nextcloud/ownCloud or fresh install:
Updated

Where did you install Nextcloud from:
From source - updated via web-updater

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

Enabled:
  - activity: 2.6.1
  - admin_notifications: 1.0.1
  - announcementcenter: 3.2.1
  - apporder: 0.5.0
  - audioplayer: 2.3.2
  - bookmarks: 0.12.1
  - bookmarks_fulltextsearch: 0.99.1
  - bruteforcesettings: 1.1.0
  - calendar: 1.6.1
  - comments: 1.3.0
  - contacts: 2.1.5
  - dav: 1.4.7
  - dicomviewer: 1.0.0
  - federatedfilesharing: 1.3.1
  - federation: 1.3.0
  - files: 1.8.0
  - files_accesscontrol: 1.3.0
  - files_downloadactivity: 1.2.0
  - files_fulltextsearch: 0.99.1
  - files_fulltextsearch_tesseract: 0.2.0
  - files_markdown: 2.0.4
  - files_pdfviewer: 1.2.1
  - files_reader: 1.2.3
  - files_retention: 1.2.0
  - files_sharing: 1.5.0
  - files_texteditor: 2.5.1
  - files_trashbin: 1.3.0
  - files_versions: 1.6.0
  - files_videoplayer: 1.2.0
  - fulltextsearch: 0.99.1
  - fulltextsearch_elasticsearch: 0.99.1
  - gallery: 18.0.0
  - gpxpod: 2.2.2
  - impersonate: 1.0.4
  - logreader: 2.0.0
  - lookup_server_connector: 1.1.0
  - metadata: 0.7.0
  - news: 12.0.4
  - nextcloud_announcements: 1.2.0
  - notifications: 2.1.2
  - oauth2: 1.1.1
  - onlyoffice: 2.0.2
  - password_policy: 1.3.0
  - polls: 0.8.1
  - previewgenerator: 1.1.0
  - provisioning_api: 1.3.0
  - rainloop: 5.1.0
  - serverinfo: 1.3.0
  - sharebymail: 1.3.0
  - spreed: 3.2.5
  - systemtags: 1.3.0
  - theming: 1.4.5
  - twofactor_backupcodes: 1.2.3
  - updatenotification: 1.3.0
  - user_ldap: 1.3.1
  - workflowengine: 1.3.0
Disabled:
  - admin_audit
  - circles
  - encryption
  - files_external
  - firstrunwizard
  - flowupload
  - survey_client
  - user_external

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "13.0.6.1",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "forcessl": true,
        "loglevel": "2",
        "logtimezone": "Europe\/Berlin",
        "theme": "",
        "overwrite.cli.url": "https:\/\/cloud.mydomain.de",
        "maintenance": false,
        "enable_previews": true,
        "preview_max_scale_factor": 10,
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\Illustrator",
            "OC\\Preview\\Movie",
            "OC\\Preview\\MSOffice2003",
            "OC\\Preview\\MSOffice2007",
            "OC\\Preview\\MSOfficeDoc",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\PDF",
            "OC\\Preview\\PhotoShop",
            "OC\\Preview\\Postscript",
            "OC\\Preview\\StarOffice",
            "OC\\Preview\\SVG",
            "OC\\Preview\\Font"
        ],
        "trusted_domains": [
            "www.mydomain.de",
            "cloud.mydomain.de"
        ],
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "lost_password_link": "https:\/\/www.mydomain.de\/selfservice",
        "auth.bruteforce.protection.enabled": false,
        "knowledgebaseenabled": true,
        "enable_avatars": true,
        "redis": {
            "port": "6379",
            "host": "***REMOVED SENSITIVE VALUE***",
            "timeout": "0"
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "data-fingerprint": "dccf22c581c820a6b4aa7fb5851c29ae",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
NO

Are you using encryption: yes/no
NO

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
SAMBA DC (Active Directory) / user_ldap

LDAP configuration (delete this part if not used)

LDAP config

+-------------------------------+-----------------------------------------------------------------------------------------------------------+
| Configuration                 |                                                                                                           |
+-------------------------------+-----------------------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 1                                                                                                         |
| hasPagedResultSupport         |                                                                                                           |
| homeFolderNamingRule          |                                                                                                           |
| lastJpegPhotoLookup           | 0                                                                                                         |
| ldapAgentName                 | CN=proxy_owncloud,CN=Proxy Users,DC=mydomain,DC=de                                                        |
| ldapAgentPassword             | ***                                                                                                       |
| ldapAttributesForGroupSearch  |                                                                                                           |
| ldapAttributesForUserSearch   | cn;sAMAccountName;surname;givenname;cn;displayname;mail                                                   |
| ldapBackupHost                |                                                                                                           |
| ldapBackupPort                |                                                                                                           |
| ldapBase                      | DC=mydomain,DC=de                                                                                         |
| ldapBaseGroups                | CN=Users,DC=mydomain,DC=de                                                                                |
| ldapBaseUsers                 | CN=Users,DC=mydomain,DC=de                                                                                |
| ldapCacheTTL                  | 60                                                                                                        |
| ldapConfigurationActive       | 1                                                                                                         |
| ldapDefaultPPolicyDN          |                                                                                                           |
| ldapDynamicGroupMemberURL     |                                                                                                           |
| ldapEmailAttribute            | mail                                                                                                      |
| ldapExperiencedAdmin          | 0                                                                                                         |
| ldapExpertUUIDGroupAttr       | sAMAccountName                                                                                            |
| ldapExpertUUIDUserAttr        | sAMAccountName                                                                                            |
| ldapExpertUsernameAttr        |                                                                                                           |
| ldapGidNumber                 | gidNumber                                                                                                 |
| ldapGroupDisplayName          | cn                                                                                                        |
| ldapGroupFilter               | (&(|(objectclass=group)))                                                                                 |
| ldapGroupFilterGroups         |                                                                                                           |
| ldapGroupFilterMode           | 0                                                                                                         |
| ldapGroupFilterObjectclass    | group                                                                                                     |
| ldapGroupMemberAssocAttr      | member                                                                                                    |
| ldapHost                      | srv2.mydomain.de                                                                                          |
| ldapIgnoreNamingRules         |                                                                                                           |
| ldapLoginFilter               | (sAMAccountName=%uid)                                                                                     |
| ldapLoginFilterAttributes     | cn;sAMAccountName;userPrincipalName;mail                                                                  |
| ldapLoginFilterEmail          | 0                                                                                                         |
| ldapLoginFilterMode           | 1                                                                                                         |
| ldapLoginFilterUsername       | 1                                                                                                         |
| ldapNestedGroups              | 0                                                                                                         |
| ldapOverrideMainServer        |                                                                                                           |
| ldapPagingSize                | 500                                                                                                       |
| ldapPort                      | 389                                                                                                       |
| ldapQuotaAttribute            |                                                                                                           |
| ldapQuotaDefault              |                                                                                                           |
| ldapTLS                       | 1                                                                                                         |
| ldapUserAvatarRule            | default                                                                                                   |
| ldapUserDisplayName           | displayname                                                                                               |
| ldapUserDisplayName2          | samaccountname                                                                                            |
| ldapUserFilter                | (&(|(objectclass=user))(|(|(memberof=CN=owncloudusers,CN=Users,DC=mydomain,DC=de)(primaryGroupID=1117)))) |
| ldapUserFilterGroups          | owncloudusers                                                                                             |
| ldapUserFilterMode            | 0                                                                                                         |
| ldapUserFilterObjectclass     | user                                                                                                      |
| ldapUuidGroupAttribute        | auto                                                                                                      |
| ldapUuidUserAttribute         | auto                                                                                                      |
| turnOffCertCheck              | 0                                                                                                         |
| turnOnPasswordChange          | 0                                                                                                         |
| useMemberOfToDetectMembership | 1                                                                                                         |
+-------------------------------+-----------------------------------------------------------------------------------------------------------+

[linuxrrze]

Reading the changelog, I suspect this change to cause that behaviour:
#10376

[nextcloud-bot]

GitMate.io thinks possibly related issues are #10932 (13.0.6), #9007 (Nextcloud 12.0.6 Design issue when uploading file), #9911 (Nextcloud upgrade to 13.0.4 Failed), #10350 (13.0.5), and #8754 (White page / Error 500 after Upgrade from Nextcloud 12.0.4 to 13 ).

[tflidd]

Nextcloud datadir contains other files

This ideally should never happen, and you should not create or delete files manually from that folder. This is why there are external local storages. If you want to do backup/snapshots, they should be stored outside the data folder.

[linuxrrze]

So I moved the snapshots to a different directory, and did a "occ files:scan --all", "occ files:cleanup" and manually removed corresponding entries from the SQL database table oc_filecache.

But the database still is about 8 times the size it was before the update.
Any hints about which database tables I also need to look at/clean up?

[MorrisJobke]

But the database still is about 8 times the size it was before the update.
Any hints about which database tables I also need to look at/clean up?

Check which tables use the most space and then we can check for further help.

[linuxrrze]

In the meantime I managed to remove all (?) references concering snapshotted files. The size of the database dump returned to normal, however the allocated space on disk is still much higher (but that's most likely a postgresql and no nextcloud topic).

[MorrisJobke]

however the allocated space on disk is still much higher (but that's most likely a postgresql and no nextcloud topic).

Correct. That is how DBs works. A vacuum command should help here.

Let's close this ticket then.