503 Error after nextcloud runs a time

[benhartwich]

Steps to reproduce

1. Install nextcloud on Debian 8 with nginx and php5.6
2. let it run a time
3. change your pc and try to reach nextcloud in chrome => 503: Service not available.

Actual behaviour

When I restart php5-fpm I can access nextcloud at chrome again. I´ve tested it with Firefox also and the same problem. It seems to be that only index.php is affected. When I run other URLs I can access nextcloud.
My php-fpm conf:

[cloud]

listen = /var/run/php5-fpm-cloud.sock
listen.backlog = 65535
listen.allowed_clients = 127.0.0.1

user = cloud
group = cloud
listen.mode = 0660
listen.owner = cloud
listen.group = www-data

pm.process_idle_timeout = 10s
pm.max_requests = 0
pm = dynamic
pm.max_children = 15
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 7

chdir = /

php_admin_value[open_basedir] = /var/kunden/cloud:/usr/share/php5:/tmp:/var/tmp:/dev/urandom:
php_admin_value[disable_functions] = dl,passthru,shell_exec,system,proc_open,popen,show_source
php_admin_value[memory_limit] = 256M
php_admin_value[max_execution_time] = 240
php_admin_value[upload_max_filesize] = 750M
php_admin_value[post_max_size] = 750M
php_admin_value[max_input_time] = 3600
php_admin_value[max_execution_time] = 3600

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Server configuration

Operating system: Debian 8

Web server: nginx 1.11.1

Database: MariaDB

PHP version: 5.6.22

Nextcloud version: 9.0.5

Updated from an older Nextcloud/ownCloud or fresh install: new install

Where did you install Nextcloud from: nextcloud.com

Signing status (Nextcloud 9.0 and above):

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.
 errors have been found.

List of activated apps:

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

The content of config/config.php:

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here
(Without the database password, passwordsalt and secret)

Are you using external storage, if yes which one: local

Are you using encryption: no

Client configuration

Browser: Chrome

Operating system: Windows 7 64bit

Logs

Web server error log

no log

Nextcloud log (data/owncloud.log)

"PHP","message":"Comments starting with '#' are deprecated in Unknown on line 1 at Unknown#0","level":3,"time":"2016-06-27T19:15:20+00:00","method":"PROPFIND","url":"\/remote.php\/webdav\/",

[LukasReschke]

That's the second report I get of this but I still wasn't able to reproduce this locally on my instances. Then again, I only do use Apache. For other NGINX users it works fine though.

Is there any chance you can get me super easy reproduction steps (e.g. a docker container) or access to your machine using SSH (https://github.com/LukasReschke.keys)? That would make it way easier for me to debug this.

[aktavor69]

Hi, its me again, we talking together few days ago

Its pretty easy to reproduce.

If you want, I reproduce my config on a VPS, I will send you credentials.

edit : email sent

[LukasReschke]

This is caused by a NGINX setting, the following is documented at https://docs.nextcloud.org/server/9/admin_manual/installation/nginx_owncloud_9x.html:

fastcgi_split_path_info ^(.+.php)(/.+)$;

This will fail hard, if somebody accesses / directly without index.php in it then $_SERVER['SCRIPT_NAME'] is set to an empty string instead of index.php. This can lead to any kind of problems and the NGINX config should probably be addressed for that.

For now I'd just recommend to remove that line from your config.

@josh4trunks @mmattel Any idea how the NGINX config could be adjusted to keep this in mind?

[benhartwich]

I´ve commented this line out, but have the same error. the owncloud client has a detailled error message: Internal Server error: The Request URI /remote.php/webdav/ did not end with the contents of PATH_INFO(i)

[josh4trunks]

@LukasReschke
The line you reference doesn't affect "SCRIPT_NAME" passed to PHP, since that variable is set in fastcgi_params. But it does affect "SCRIPT_FILENAME"

A request for "/" would be matched by the / block, it does not match the PHP block with the line you reference.
It would then be rewritten to "/index.php" > then match the PHP block, which would set the "SCRIPT_FILENAME" to "/index.php"

[josh4trunks]

@benhartwich if you comment out that line, alot of aspects of owncloud won't work.

@LukasReschke I don't think you should tell people to remove that line =P

[LukasReschke]

https://github.com/nextcloud/server/blob/stable9/lib/base.php#L518 calls IRequest::getScriptName(), this function reads $_SERVER['SCRIPT_NAME']. With the documented NGINX config SCRIPT_NAME is empty when opening / instead of /index.php. (just do a var_dump($_SERVER);exit(); in your index.php)

Removing that line from the config sets the proper SCRIPT_NAME. I don't have much experience with NGINX but this seems just wrong. (expected behaviour would be to get "index.php")

[josh4trunks]

@LukasReschke A request for "/" would not be passed to PHP. It would first be rewritten to "/index.php"

Please see #237 (comment)

[LukasReschke]

@josh4trunks To illustrate the problem, I have created a docker. It is based on the config from https://docs.nextcloud.org/server/9/admin_manual/installation/nginx_examples.html (which is basically a 1:1 copy of the ownCloud docs page)

To test it please execute:

git clone https://gist.github.com/069d68d341254e265a143a9a3385c52c.git issue-237
cd issue-237/
docker build -t nextcloud .
docker run -t -i nextcloud /bin/bash

Inside the container start PHP FPM and NGINX:

/etc/init.d/nginx start && /etc/init.d/php5-fpm start

Now cURL /index.php and / directly:

root@63c097aa0c98:/# curl http://localhost/
string(0) ""
root@63c097aa0c98:/# curl http://localhost/index.php
string(10) "/index.php"

As you can see, the index.php is invoked in both but the proper SCRIPT_NAME is only set for the one where the index.php is appended. That's actually the problem that I'm pointing out here.

From a software perspective we do expect to get a proper $_SERVER['SCRIPT_NAME'] for the / URL as well.

[josh4trunks]

@LukasReschke Can you post/link to the exact nginx config and fastcgi_params so I can take a look.
If it is exactly as the docs, "/" would be internally rewritten to "/index.php" and SCRIPT_NAME would be "/index.php".

Can you also try replacing the index.php with this to confirm SCRIPT_NAME is being set incorrectly.

<?php
phpinfo();
?>

I can test this as well tonight. Currently running ownCloud but if you are correct this issue would crop up before touching the PHP code.

####EDIT####
Looking at your docker file, looks like you did what I'm thinking. Can you post your fastcgi_params

[LukasReschke]

You can find the NGINX config file at https://gist.github.com/069d68d341254e265a143a9a3385c52c, the fastcgi_params are the defaults from Debian and can be found at https://gist.github.com/LukasReschke/65c3cb807e5e4be8a21dcfd26d19d6de. This is a new Docker container that can be easily setup locally by performing the steps from #237 (comment) :)

Can you also try replacing the index.php with this to confirm SCRIPT_NAME is being set incorrectly.

Output of http://localhost/ is at https://rawgit.com/LukasReschke/1937e99d78a85216f917c9b769215ce4/raw/9d512fc9bfac73a9400e1d5900945d72d4590ee3/gistfile1.html

Output of http://localhost/index.php is at https://rawgit.com/LukasReschke/c386766a72a37aa0ce2dfb889beaed42/raw/80534db1501abcb3676ed539df79286b3e513bca/phpinfo.html

You also see that _SERVER["SCRIPT_NAME"] is only set correctly in the later case.

[josh4trunks]

Ok, I think I see what if going on, but I don't understand why, and I can't reproduce with my FreeBSD system. It looks like a bug with nginx in this case.

$fastcgi_script_name is being unset during nginx's rewrite.
A workaround is switch the order of include fastcgi_params; to after fastcgi_split_path_info ^(.+\.php)(/.+)$;

[josh4trunks]

Even in the case of /index.php $SERVER_NAME is blank. Can you confirm fastcgi_params is being included? Maybe hardcode SERVER_NAME to something and make sure it does show.

Ohh, and no docker for me. No access to my linux desktop at the moment, only my freebsd servers.

[LukasReschke]

It is. If I change it in the fastcgi_params file to fastcgi_param SERVER_NAME "Hardcoded"; the value "Hardcoded" is displayed in the phpinfo output.

[aktavor69]

@LukasReschke if you need, you can send ssh axx from my test server to @josh4trunks

[josh4trunks]

ok, I think this is a bug with the nginx being used. My nginx changes the value of $fastcgi_params after the rewrite, while this build look like it is unsetting or not changing it.

[LukasReschke]

root@0cb051da90b7:/# nginx -v
nginx version: nginx/1.11.1

[josh4trunks]

I can't SSH at the moment. Could debug more deeply tonight.

Here's a test..
nginx config

location = /test {
  rewrite ^ /test.php$uri;
}
location = /test.php {
  fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}

test.php

<?php
var_dump($_SERVER['SCRIPT_NAME']);

[josh4trunks]

I'm using version 1.8. I can update to 1.10 and see if I can reproduce this behavior. But the workaround of switching the order of the two lines is probable safest to solve this in all cases.

[josh4trunks]

I think I have been misunderstanding this.
Based on http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html $fastcgi_script_name is set by the request URI.
I never hit this problem because I have fastcgi_index set to "index.php" but this is not default.

My suggestion is we move switch the order like so

        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        include fastcgi_params;

EDIT Please see this #237 (comment)

[aktavor69]

Same issue with this order

[josh4trunks]

Hmm, I guess we should split regarless of if an actual file is being requested.
Please try this...

fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;

[aktavor69]

Seems working :D

[MorrisJobke]

@josh4trunks Could we add this to the documentation? Or is this something special to the environment?

[josh4trunks]

We should make this change. In default environments this is needed since we rely on rewrite now. It only doesn't affect some if they have something like fastcgi_index set to index.php, while this is common, it is not default.

Change is from...

include fastcgi_params;
fastcgi_split_path_info ^(.+\.php)(/.+)$;

to...

fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;

This needs to be applied to both
https://docs.nextcloud.org/server/9/admin_manual/installation/nginx_owncloud_9x.html#nextcloud-in-the-webroot-of-nginx
https://docs.nextcloud.org/server/9/admin_manual/installation/nginx_owncloud_9x.html#nextcloud-in-a-subdir-of-nginx

[MorrisJobke]

@josh4trunks nextcloud/documentation#40 should do the trick then ;)

[aktavor69]

@MorrisJobke Nop, its not right.

You forgot to remove

fastcgi_split_path_info ^(.+\.php)(/.+)$;

and add instead

fastcgi_split_path_info ^(.+\.php)(/.*)$;

[MorrisJobke]

@MorrisJobke Nop, its not right.

@aktavor69 Thanks a lot. I didn't spot that :( I fixed it :)

[MorrisJobke]

This should be fixed with the new Nginx config

[Ardakilic]

I had the same issue after migrating from owncloud,

Simply switching the order of these two like this, as described above, fixed the problem for me:

fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;

I'm using Debian 8.6, PHP7.0, MariaDB, Nginx 1.10.1. All manual install.

[techc0de]

Hi,

I don't meant to re-open this thread, but the NC admin manual have incorrect order of those above two lines. The above solution is verified.
FYI: this error only occur on Android devices that use NextCloud app.

Please update the admin manual.

Thanks.

[MorrisJobke]

Hi,

I don't meant to re-open this thread, but the NC admin manual have incorrect order of those above two lines. The above solution is verified.
FYI: this error only occur on Android devices that use NextCloud app.

Please update the admin manual.

Our manual is also open source. 😉 Nevertheless I fixed it - see nextcloud/documentation#371