14.0.5 master could not get Loop data

[bjornoleh]

I changed to 14.0.5 master today for two NS instances fed with Loop data and CGM via Dexcom Bridge. CGM data was displayed, but no new Loop data was shown in the main screen after updating, even after >20 minutes. The Loop pill (FreeAPS) reported the age of the old Loop data

I did not yet check reports.

Reverting to Dev 14.0.3 made Loop data load as usual.

[bjornoleh]

@sulkaharo I changed one of the NS instances back to 14.0.5 master again, with the same result. Some Loop data are shown in Loop, as far as I can see this is temp basals and boluses, and the text in the IOB pill. The IOB is outdated.

[bjornoleh]

It looks pretty much the same in the reports:

[bjornoleh]

Reverting to 14.0.4 Master: IOB and Loop/FreeAPS data are again up to date

[sulkaharo]

That's very curious. Is there anything in Nightscout logs that acts as a hint on what's going on? None of the changes in the update are Loop specific, all unit tests work and I'm seeing data updates to all collections going through as expected. Basically - I have no idea where the issue might be unless I get more data

[bjornoleh]

Which logs should I look at? søn. 27. sep. 2020, 20:07 skrev Sulka Haro <notifications@github.com>:

…

That's very curious. Is there anything in Nightscout logs that acts as a hint on what's going on? None of the changes in the update are Loop specific, all unit tests work and I'm seeing data updates to all collections going through as expected. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEZWM6MG3HBEWEUBIBAOGLSH55NXANCNFSM4R3UZIFQ> .

[sulkaharo]

The one Nightscout outputs - if you're on Heroku, check Papertrail

[bjornoleh]

At least there are no new entries in devicestatur collection (mLab) here.

Yes, I'm on Heroku. In Papertrail there is this, some problem with the recent change with /pebble possibly? We have set up both NS instances with AUTH_DEFAULT_ROLES: denied, could there be some issue with the tokens?

Sep 27 12:02:16 xxxx app/web.1 DENIED: 158.xxx.xxx.63 no-token api:pebble,entries:read
Sep 27 12:02:17 xxxx heroku/router at=info method=GET path="/pebble?count=47" host=xxxx.herokuapp.com request_id=ca56d7a6-0ec3-4480-a579-a13612256237 fwd="158.xxx.xx.63" dyno=web.1 connect=1ms service=7ms status=401 bytes=758 protocol=https
Sep 27 12:02:17 xxxx app/web.1 DENIED: 158.xxx.xx.63 no-token api:pebble,entries:read
Sep 27 12:02:17 xxx heroku/router at=info method=GET path="/api/v2/properties" host=xxxx.herokuapp.com request_id=dfd889a8-649f-4f25-a722-7aaec15ce583 fwd="158.xxx.xx.63" dyno=web.1 connect=1ms service=59ms status=200 bytes=2730 protocol=https

[sulkaharo]

The /pebble change only changed how the endpoint gets the configured units and did not affect the api permissions. That log at least shows you have some app trying to read the api but it's being denied. If this is Loop, it'll surely cause issues.

[bjornoleh]

Apparently it does.

Edit: And yes, this is Loop (FreeAPS) tryng to upload data to the devicestatus collection in mLab (I guess I'll have to migrate to Atlas soon...)

The treatments collection is updated as it should, from Loop. But devicestatus is not. This includes IOB, COB etc, and most noticeable, the predicted BG data. Ill paste a sample below, the last one that got in before going to 14.0.5.

I tried changing AUTH_DEFAULT_ROLES to readable and admin, but no difference to be seen. But that would probably not make a difference here anyways.

"_id": {
    "$oid": "5f70dc5d1319bc000436874c"
},
"override": {
    "timestamp": "2020-09-27T18:39:24Z",
    "active": false
},
"pump": {
    "suspended": false,
    "model": "Eros",
    "secondsFromGMT": 7200,
    "pumpID": "xxxxxx",
    "clock": "2020-09-27T18:39:24Z",
    "manufacturer": "Insulet",
    "bolusing": false
},
"uploader": {
    "battery": 73,
    "name": "iPhone",
    "timestamp": "2020-09-27T18:39:24Z"
},
"loop": {
    "cob": {
        "timestamp": "2020-09-27T18:35:00Z",
        "cob": 8.045095689807003
    },
    "version": "2.1",
    "name": "FreeAPS",
    "timestamp": "2020-09-27T18:39:24Z",
    "recommendedBolus": 0.15,
    "iob": {
        "timestamp": "2020-09-27T18:35:00Z",
        "iob": 0.32186044857546825
    },
    "predicted": {
        "values": [
            77,
            77,
            77,
            76,
            76,
            75,
            75,
            74,
            74,
            74,
            75,
            75,
            76,
            77,
            78,
            79,
            80,
            81,
            82,
            83,
            84,
            85,
            87,
            88,
            89,
            90,
            91,
            93,
            94,
            95,
            96,
            97,
            98,
            99,
            100,
            101,
            102,
            103,
            104,
            105,
            106,
            107,
            108,
            108,
            109,
            109,
            110,
            110,
            111,
            111,
            111,
            111,
            111,
            112,
            112,
            112,
            112,
            112,
            112,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113,
            113
        ],
        "startDate": "2020-09-27T18:39:11Z"
    }
},
"device": "loop://iPhone",
"created_at": "2020-09-27T18:39:24Z"

}

[bjornoleh]

@sulkaharo Let me know if I can provide any other info or do some tests before I roll back to a working version for the night.

[sulkaharo]

Can you try just removing AUTH_DEFAULT_ROLES for a sec to see if this is a permission issue?

[sulkaharo]

Ok wait a second - how did you get that device status record? The _id record on it doesn't look right and trying to insert that would fail

[bjornoleh]

@sulkaharo Do you mean this bit?

"_id": {
"$oid": "5f70dc5d1319bc000436874c"

It is just copied from mLab

It looks the same if I find an older one:

{
"_id": {
"$oid": "5ef8ba89aa71a40004c34fbb"
},
"loop": {
"timestamp": "2020-06-28T15:43:05Z",

[bjornoleh]

Removing AUTH_DEFAULT_ROLES do not seem to make a difference in NS or mLab, but I don't think I see the error message in Papertrail any more...

The error seems to have gone away when removing AUTH_DEFAULT_ROLES, but I am not quite sure that this is actually related to the issue at hand. It could have been a coincidence, as the same error repeats itself as far back as the Papertrail log is displayed (two days).

[andrew-dixon]

Having the same issue here, just going back to 14.0.2 to see if that fixes it, as that was the version I was on before I updated earlier today.

[andrew-dixon]

@sulkaharo I've downgraded 14.0.2 and it is working again.

[sulkaharo]

Did you see any errors in the Nightscout log? Looks like some some subtle little change happened that affects only Loop and this is practically impossible for me to debug. Ping @ps2

[bjornoleh]

Not sure if his is any help?

WS: Connecion
WS: Disconnected

[bjornoleh]

I'll roll back to something that is working now, but let me know if I can test anything tomorrow.

[sulkaharo]

Right, so if someone can PM me your instance URL in Discord, I can try to take a peek at the data to see if I can find a lead to what's the cause.

[sulkaharo]

@bjornoleh the WS lines are about websocket connections, ie the Nightscout web app connecting to the backend. Loop uses the REST API, so the interesting lines would be either error stack traces or related to an app accessing /api/v1 URLs

[miki3421]

Same for me. Following..

[sulkaharo]

The above linked PR should fix this. @bjornoleh is testing the fix

[sulkaharo]

(Looks like the reason this wasn't found in testing is, Loop does batch uploads if the uploads have fallen behind. This only happens in situations like updating a production site. AFAIK testing is usually done by updating a site and pointing Loop to the updated site, which causes no downtime to be observed by Loop.)

[sulkaharo]

This is now fixed in master as part of 14.0.6, please update your site to get the fix