Offline federated share causes slow down and cannot be removed

[blizzz]

Steps to reproduce

1. Receive a federated share and accept it
2. The federated share's host goes offline
3. In the web interface go to the Files app → Slow (everything turns slow)
4. Try to "Unshare" the federated share

Expected behaviour

1. (Ideally) Federated shares do not affect the operation of your Nextcloud. Might be tricky in fact, if solvable at all.
2. Removing the federated share should be dependent on whether the remote host is online and operating properly

Actual behaviour

1. Nc is very slow, also when using unrelated apps, e.g. News.
2. A 503 is received after a long time. Request and Respons headers and data:

DELETE /remote.php/webdav/$FederatedShareName HTTP/1.1
Host: my.nextcloud.server
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,en-US;q=0.7,de;q=0.3
Accept-Encoding: gzip, deflate, br
X-Requested-With: XMLHttpRequest
requesttoken: $requesttoken
Cookie: nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; 50a41faf46708=uqc39q49cop0umd29mo6k34s85; oc_sessionPassphrase=$sessionPassphrase; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; nc_username=arthur; nc_token=$token; nc_session_id=$session_id
DNT: 1
Connection: keep-alive

HTTP/1.1 503 Service Unavailable
Date: Tue, 13 Dec 2016 10:56:04 GMT
Server: Apache/2.4.18 (Ubuntu)
strict-transport-security: max-age=15768000; includeSubDomains; preload
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Robots-Tag: none
X-Frame-Options: SAMEORIGIN
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Content-Length: 238
Connection: close
Content-Type: application/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
  <s:message>Storage is temporarily not available</s:message>
</d:error>

The Unshare-issue was also present in Nc 10, but it did not get the performance hit.

(Removing the federate share manually from share_external table fixes the slow-down issue 🙊 )

Server configuration

Operating system: Ubuntu 16.04

Web server: Apache2

Database: MySQL 5.7

PHP version: 7.0.8

Nextcloud version: 11.0.0

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 10

Where did you install Nextcloud from: tarball

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: FF 50

Operating system: Antergos

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"cFtBlcCr7VOs4WDouuYj","remoteAddr":"92.200.49.78","app":"webdav","message":"Exception: {"Message":"HTTP\/1.1 503 Storage is temporarily not available","Exception":"Sabre\DAV\Exception\ServiceUnavailable","Code":0,"Trace":"
#0 \/var\/www\/minion\/ovin-11.0.0.9\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php(178): OCA\DAV\Connector\Sabre\ObjectTree->getNodeForPath('$FederatedShareName')\
#1 \/var\/www\/minion\/ovin-11.0.0.9\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(287): Sabre\DAV\Tree->delete('$FederatedShareName')\
#2 [internal function]: Sabre\DAV\CorePlugin->httpDelete(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\
#3 \/var\/www\/minion\/ovin-11.0.0.9\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\
#4 \/var\/www\/minion\/ovin-11.0.0.9\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\Event\EventEmitter->emit('method:DELETE', Array)\
#5 \/var\/www\/minion\/ovin-11.0.0.9\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\
#6 \/var\/www\/minion\/ovin-11.0.0.9\/apps\/dav\/appinfo\/v1\/webdav.php(60): Sabre\DAV\Server->exec()\
#7 \/var\/www\/minion\/ovin-11.0.0.9\/remote.php(165): require_once('\/var\/www\/minion...')\
#8 {main}","File":"\/var\/www\/minion\/ovin-11.0.0.9\/apps\/dav\/lib\/Connector\/Sabre\/ObjectTree.php","Line":162,"User":"arthur"}","level":4,"time":"2016-12-13T10:58:32+00:00","method":"DELETE","url":"/remote.php/webdav/$FederatedShareName","user":"arthur","version":"11.0.0.9"}

Browser log

see on top

cc @nextcloud/sharing

[schiessle]

In the web interface go to the Files app → Slow (everything turns slow)

After the federated share is established it is a webdav mount so it is all about how we handle external storage in case a source is offline, nothing federated sharing specific.

Removing the federated share should be dependent on whether the remote host is online and operating properly

Don't understand this part. Unsharing should always work. Does it fail if the source is offline?

[schiessle]

Regarding the slowness. How did you tested it? I just set up two servers, shared from server1 to server2 and then changed the root directory of server1 which results in a 404. I don't see any negative impact on the performance. Can browse the rest of the files as before.

Indeed, unsharing doesn't work... I will debug this one.

[blizzz]

Don't understand this part. Unsharing should always work. Does it fail if the source is offline?

Exactly

[blizzz]

Regarding the slowness. How did you tested it? I just set up two servers, shared from server1 to server2 and then changed the root directory of server1 which results in a 404. I don't see any negative impact on the performance. Can browse the rest of the files as before.

Once upon a time I got a federated share from @MorrisJobke (private instance) to my private instance. Then, his server with just offline, I think, and since 11 it was behaving slowing (running into timeouts). With moving it on the server, you will not receive a timeout, the 404 comes pretty much immediately.

[schiessle]

The unsharing stuff seems to be quite complicated. It fails deep in the SabreDAV code so that the delete is never executed and therefore we never reach the point where we could perform a unshare.

[blizzz]

Yup.

[blizzz]

A user on IRC ran into the same issue… and suggesting to blindly delete a row from the DB to get a properly working Nextcloud back feels very very bad 🙊

[Nosion]

And it worked, removing the one row I had in oc_share_external fixed the issue! Thanks @blizzz

[JarbyDev]

Same problem here, i trying to remove a federated share, but i get a error message error deleting. Nextcloud 12.

[ghost]

I have this same problem. I had to remove my federation add folder.

This really needs to be a better handled error. There's no errors on federation downtime in logs....

A few things which really need to be done, imo (and I may be able to assist with writing some of these, depending on my time):

• Proper logging for slow federated shares (perhaps a default of any response > 5 sec)
• More graceful handling of remote share down/up
• currently, a remote share outage causes extremely slow responses to the entire nextcloud interface. We need either:
  • Faster timeouts for share checking
  • multithreaded behavior, allowing shares to come online in the background without blocking the loading of each gui element for 30-60 seconds.

[PeterEdens]

Just experienced this problem also, went through php, nginx and reverse proxy looking for the problem and finally saw in the nextcloud log a reference to the share not responding. This is a bad bug that can deny access to a server so should be a higher priority to fix.

[HNieme]

Similar problem here:
I had an old owncloud and the new nextcloud.
I took the old owncloud offline, resulting in the federated files not being able to be shared again:
"Das Weiterverteilen ist nicht erlaubt" (german, meaning "Resharing is not allowed").
Found out that deleting is not possible, too.

DELETING the rows in the database of oc_share_external worked, external shares are now gone.

I am running the nextcloud Docker-container.

[MorrisJobke]

Duplicate of #7001 and fixed by #7485