PayloadTooLargeError: request entity too large #7129

old-square-eyes · 2021-10-12T08:27:21Z

Describe the bug
NS stops displaying treatments. This includes OpenAPS insulin, temp basal rates. Visually this propagates to NSClient also.

ns-looplog on OpenAPS shows the following error - probably a HTML page being served up by the NS API when it shouldn't be...

Refreshed carbhistory; COB: 0
{"batteryVoltage":3647,"battery":45}
{"BG":92,"IOB":0.442,"rate":null,"duration":null,"units":null}
Latest NS treatment: 2021-10-12T06:08:32.000Z
<html>
  <head>
    <meta charset='utf-8'>
    <title>PayloadTooLargeError: request entity too large</title>

Looking at Atlas DB, all writes cease, as if NS has locked up. Dropping the treatments collection temporarily fixes things, but the lock up returns.

I have 4 clients. This happens on Master and Dev NS.

2 x OpenAPS edison rigs
xDrip (uploading BG)
NSClient for monitoring / care portal.

The text was updated successfully, but these errors were encountered:

old-square-eyes · 2021-10-12T08:32:18Z

I saw this, #5704 but am guessing since I tried Dev it either hasn't been implemented everywhere or my issue is slightly different.

old-square-eyes · 2021-10-12T10:48:40Z

Furthermore I just noticed my Autotune is failing to pick up manual pump basal changes (I recently had to drop the rate in the evening significantly because of frequent lows). This adjustment is not reflected in my loops. I ran Autotune manually and got the following error...

Query: https://mysite.herokuapp.com treatments.json find%5Bcreated_at%5D%5B%24gte%5D=2021-10-10T06:00+13:00&find%5Bcreated_at%5D%5B%24lte%5D=2021-10-12T18:00+13:00
-rw-r--r-- 1 root root 3 Oct 12 23:43 ns-treatments.2021-10-11.jsonoref0-autotune-prep   ns-treatments.2021-10-11.json profile.json ns-entries.2021-10-11.json profile.pump.json > autotune.2021-10-11.json
/root/src/oref0/bin/oref0-autotune-prep.js:107
        return console.error("Warning: could not parse "+glucose_input", e);
   ^^^^^^
SyntaxError: Invalid or unexpected token
    at Module._compile (internal/modules/cjs/loader.js:723:23)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
Could not run oref0-autotune-prep ns-treatments.2021-10-11.json profile.json ns-entries.2021-10-11.json

It's hard to tell but it seems like the NS API could be responding incorrectly perhaps?

cat ns-treatments.2021-10-11.json []

File is an empty list :(

sulkaharo · 2021-10-12T13:41:32Z

The error implies your rig is trying to upload a very large chunk of JSON data in one go and it's overflowing the server. Looks like we can bump up the buffer size for this. The autotune error looks like something else

bewest · 2021-10-12T16:41:37Z

I've been doing some support work on a similar issue. It turns out the default request body size set by expess's bodyParser middleware is 100Kb. With openaps in particular, a few days' worth of backlogged data can be larger than this. I've had success pairing down the data to upload less at one time, but this can be onerous for end users. It is necessary to set some sort of limit, but should we or the end user be able to set some sort of policy, potentially at runtime?

I see common options for 5Mb - 50Mb. Some clients, such as xdrip, are able to circumvent the limit by gzipping their payloads. I suspect 5mb would be enough and would be happy to put together a PR for this.

old-square-eyes · 2021-10-13T00:17:02Z

Thanks both for your responses. @bewest I'm happy to test a feature branch when the time comes.

bewest · 2021-10-13T04:49:46Z

@old-square-eyes, can you confirm the size of the files in the upload subdirectory? Something like ls -alh upload. It looks like Nightscout should allow uploads up to 50Mb to the treatments api, and 1Mb otherwise. I do see some potential issues with a deprecation and will double check, but it would be good to get an idea of the how large the files you are uploading are.

If you are not sure, you can schedule some time with me to help figure it out.

old-square-eyes · 2021-10-13T04:59:13Z

@bewest am I in the right dir?

root@apsgreen:~/myopenaps# ls -alh upload
total 228K
drwxr-xr-x  2 root root 4.0K Oct 13 17:55 .
drwxr-xr-x 11 root root 4.0K Oct 10 13:40 ..
-rw-r--r--  1 root root 215K Oct 13 17:55 latest-treatments.json
-rw-r--r--  1 root root 1.3K Oct 12 22:18 recent-pumphistory.json

and

root@apspiw:~/myopenaps# ls -alh upload
total 12K
drwxr-xr-x  2 root root 4.0K Oct 13 16:57 .
drwxr-xr-x 11 root root 4.0K Sep 12 17:25 ..
-rw-r--r--  1 root root    0 Oct 13 17:56 latest-treatments.json
-rw-r--r--  1 root root 1.4K Oct 12 18:48 recent-pumphistory.json

bewest · 2021-10-13T06:00:38Z

Yes, perfect, @old-square-eyes. Thanks for the information. Your file is 215K, which is more than the 100K limit being enacted currently. I'm currently testing with a similar file, which contains 30k records in it. My wip/bewest/request-body-limit feature branch should help address this issue: #7131

old-square-eyes · 2021-10-13T22:47:49Z

@bewest is this ready for me to test? Or is it a work in progress (as wip might suggest)

old-square-eyes · 2021-10-14T04:43:59Z

@bewest I tried your feature branch and it's been working for 6 hours so far. Thank you my friend! This was a great source of distress and it means a lot for you to have jumped on this. Frankly I have no idea why I seem to be one of only a few with the issue. I have no idea why I had a large upload. The rig in question is a daily driver so shouldn't have gone offline for any length of time necessitating a big upload when returning online.

old-square-eyes · 2021-10-14T04:46:57Z

I just tried a manual run and Autotune still fails. I am currently using NS tokens. I might switch to API key to see if that resolves the issue. I recall @sulkaharo once put in a patch to allow tokens as secrets, which worked for me... I just thought OpenAPS Dev fixed that. But maybe they didn't. Will report back. Thanks again both.

old-square-eyes · 2021-10-17T22:04:08Z

Autotune is working again per openaps/oref0#1410

Completely separate issue.

bewest · 2021-10-19T19:24:32Z

@old-square-eyes: Thanks for all the feedback on this issue! It's been very helpful. Closing for now, as the patch is included in dev branch towards next release.

old-square-eyes mentioned this issue Oct 12, 2021

Error 413 #7104

Closed

bewest closed this as completed Oct 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PayloadTooLargeError: request entity too large #7129

PayloadTooLargeError: request entity too large #7129

old-square-eyes commented Oct 12, 2021 •

edited

Loading

old-square-eyes commented Oct 12, 2021

old-square-eyes commented Oct 12, 2021 •

edited

Loading

sulkaharo commented Oct 12, 2021

bewest commented Oct 12, 2021

old-square-eyes commented Oct 13, 2021

bewest commented Oct 13, 2021

old-square-eyes commented Oct 13, 2021 •

edited

Loading

bewest commented Oct 13, 2021

old-square-eyes commented Oct 13, 2021

old-square-eyes commented Oct 14, 2021

old-square-eyes commented Oct 14, 2021

old-square-eyes commented Oct 17, 2021 •

edited

Loading

bewest commented Oct 19, 2021

PayloadTooLargeError: request entity too large #7129

PayloadTooLargeError: request entity too large #7129

Comments

old-square-eyes commented Oct 12, 2021 • edited Loading

old-square-eyes commented Oct 12, 2021

old-square-eyes commented Oct 12, 2021 • edited Loading

sulkaharo commented Oct 12, 2021

bewest commented Oct 12, 2021

old-square-eyes commented Oct 13, 2021

bewest commented Oct 13, 2021

old-square-eyes commented Oct 13, 2021 • edited Loading

bewest commented Oct 13, 2021

old-square-eyes commented Oct 13, 2021

old-square-eyes commented Oct 14, 2021

old-square-eyes commented Oct 14, 2021

old-square-eyes commented Oct 17, 2021 • edited Loading

bewest commented Oct 19, 2021

old-square-eyes commented Oct 12, 2021 •

edited

Loading

old-square-eyes commented Oct 12, 2021 •

edited

Loading

old-square-eyes commented Oct 13, 2021 •

edited

Loading

old-square-eyes commented Oct 17, 2021 •

edited

Loading