Resetting cross-signing can get completely broken if you abort a halfway failed attempt

[bwindels]

While testing retrying account data requests during cross-signing reset, I managed to get reset cross-signing in a completely broken state. Even on a fresh login with that account, I can't reset the cross-signing keys anymore.

During testing, I purposefully made set account data requests fail, and I refreshed while not having completed the reset operation. In any case, refreshing the app, or having requests fail should not leave you in a state where cross-signing is just completely broken and there is no way to fix it. I just get this dialog when trying:

When trying to verify somebody in this state, I get a passphrase dialog, and when entering my passphrase nothing happens with the following error in the console:

CrossSigning.js?c1e7:122 Uncaught (in promise) Error: Key type master from getCrossSigningKey callback did not match
    at CrossSigningInfo.getCrossSigningKey (CrossSigning.js?c1e7:122)
    at async Crypto.checkOwnCrossSigningTrust (index.js?89dd:1314)
    at async Crypto.bootstrapSecretStorage (index.js?89dd:719)
    at async accessSecretStorage (CrossSigningManager.js?bc8a:238)
    at async enable4SIfNeeded (verification.js?ee94:35)
    at async verifyUser (verification.js?ee94:117)

The security settings show this:

This was on a local synapse that I last updated on Monday Apr 20, last commit f5ea8b48bd57c19dae126a5cc631dc79cf5ce332.

This is the account data of the broken account:

{
  "m.secret_storage.key.NqawOlJcxXqeVRWYcgV9h6SMb89TNOe0": {
    "type": "m.secret_storage.key.NqawOlJcxXqeVRWYcgV9h6SMb89TNOe0",
    "content": {
      "algorithm": "m.secret_storage.v1.curve25519-aes-sha2",
      "passphrase": {
        "algorithm": "m.pbkdf2",
        "iterations": 500000,
        "salt": "y0UEjkoLynttHNQAAStmEe5aQKmit0IW"
      },
      "pubkey": "R9kN1CKC6uo8PicfPl7l+PmvxJRirLLVzDmBR3/hd08",
      "signatures": {
        "@bruno-cs2:localhost": {
          "ed25519:j6vzDKIjYJFWxeWVIOBYi3Z0NZYpchgzrbx0vKYOWIA": "9bnkJVfnoM6OeVdXBWo+TX36Q8lAm7Y6ynOfAqPuv46L/sYlzknUT20fazyPyHsehxKHQdQk618YOcfP49YFBw"
        }
      }
    },
    "unsigned": {}
  },
  "m.secret_storage.default_key": {
    "type": "m.secret_storage.default_key",
    "content": {
      "key": "UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz"
    },
    "unsigned": {}
  },
  "m.secret_storage.key.UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz": {
    "type": "m.secret_storage.key.UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz",
    "content": {
      "algorithm": "m.secret_storage.v1.aes-hmac-sha2",
      "passphrase": {
        "algorithm": "m.pbkdf2",
        "iterations": 500000,
        "salt": "y0UEjkoLynttHNQAAStmEe5aQKmit0IW"
      },
      "iv": "RWyi8OHhl+0YdWAc1c1d3g==",
      "mac": "g5tEycHZgFMRCA5n2v5dgKeGQDHD0XoKeCKmDBOa1nc=",
      "signatures": {
        "@bruno-cs2:localhost": {
          "ed25519:j6vzDKIjYJFWxeWVIOBYi3Z0NZYpchgzrbx0vKYOWIA": "uDTSYBEsBGk7FFDg3oMz4r2tQ7gheBCIohdYBeKH5jfvyph7t0edTtrZlG3jHywFE58XS+liFqYRQ0uWHJBBDA",
          "ed25519:J+RSIygjIzFbtvmrnJ6hplo/wtbd/gVJsw0Bg24B/Kc": "92dB/bLhnrgzK7rsYh+5fh8m55CGGVmvReLSKftXs/UMXRfVNh/BsnujGhKJgkH+j+lem838rlmlsZC3gphaDA",
          "ed25519:o6KENMS74YIKmQdyFuRkxMYBhxgvnyn1FScHAEuCuSk": "p7SAlqZC+BvCzSIMODWr1DhtTGedln7pLorqqJQnDlD9x7nNtagEtYMDs8B+lmNPtZ23cw5huE0QpRSnQJ2ZCA",
          "ed25519:mmjnpW61nos/UMdbnB5xDA4LYq3uWhG2UWm8sTwzmKs": "BfQwWAcwh5hqF10aylQXyMDqFdAVR6VAnlBwgmlCz+JCoqaiBC42k4AEXAuaQJLmKlFNkCT+/D+a7HLLBfJTBA"
        }
      }
    },
    "unsigned": {}
  },
  "m.cross_signing.master": {
    "type": "m.cross_signing.master",
    "content": {
      "encrypted": {
        "UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz": {
          "iv": "eA48yeLZnb474LvIrsPWyg==",
          "ciphertext": "Inep5o8uPSJ3JSQtdvn5llt8LHQb7EJBg50gCbbMCjZ3mM5hDzjGyqBIme4=",
          "mac": "qQyC1CIs/LBfsPS/UrhNf5Wt8F+aCxkgiO1v+tOoLVM="
        }
      }
    },
    "unsigned": {}
  },
  "m.cross_signing.self_signing": {
    "type": "m.cross_signing.self_signing",
    "content": {
      "encrypted": {
        "UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz": {
          "iv": "9/rBfZ2ulK0tkhDmiR4IkQ==",
          "ciphertext": "BCWiQJK/IThOHvokatv3P7A6/D78uHb5UOKBykZkFiDjluixuOCLIzDWBE8=",
          "mac": "ny7Fdim7aqAkdJG95Ks4z3PPygoa3dkf3lIbFneaLhE="
        }
      }
    },
    "unsigned": {}
  },
  "m.cross_signing.user_signing": {
    "type": "m.cross_signing.user_signing",
    "content": {
      "encrypted": {
        "UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz": {
          "iv": "GQU2r9LDQAIXTr3wxSzo8Q==",
          "ciphertext": "+EGISntCnPi9PFrjrDGKxElkKDH3G8sPzl0QuK0r+VqexWOcB/9s4AfXDjI=",
          "mac": "976KULDLCXApHmCjUocdKANzt2G8NhEWSmoOALxvGJY="
        }
      }
    },
    "unsigned": {}
  },
  "m.megolm_backup.v1": {
    "type": "m.megolm_backup.v1",
    "content": {
      "encrypted": {
        "UDEgkBfKmKP9IZIgeYkb3GqcpSqSDlUz": {
          "iv": "Qa/pR5CxKEUfupnCQYBipA==",
          "ciphertext": "zviX2Yf4nAs5lN34ZjsJwmGEgGZqF13GY8YW4vJfhEQ0cGH4AP1PofpU+bc=",
          "mac": "0/TASoY80XD9zT5kkdNo2jukQ8lA+6msUedjldVhReQ="
        }
      }
    },
    "unsigned": {}
  },
}

---

@jryans: Edited to remove unrelated account data

[jryans]

Agreed this is not good, as reset should always work. It is a bit artificial, but still seems like it could happen in reality too if the connection drops out. Nice to have a fix for release if possible, but not a blocker.

[superboum]

I encountered this error on my production instance. What should I do now?

[jluebbe]

I seem to have triggered this issue on riot.im. Is there any way to recover?

[dbkr]

Looks like the problem here is that if you have a key backup,CreateSecretStorageDialog will try to migrate it even if it's supposed to be force-resetting.

[dbkr]

No wait, it skips straight to PHASE_PASSPHRASE if the force prop is true.

[dbkr]

I tried to repro this by blocking the upload of the master key and then refreshing while it spins retrying to upload it, but resetting still works fine after refreshing and unblocking.

I also can't see any way the bootstrap function would print, "Cross-signing private keys found in secret storage" when the reset button was pressed: it should never enter that block. Were you definitely pressing the bootstrap button rather than the reset button?

Anyone else, if you think you're hitting this, please submit your logs (settings -> help & about -> submit debug logs).

[bwindels]

Looks like the problem here is that if you have a key backup,CreateSecretStorageDialog will try to migrate it even if it's supposed to be force-resetting.

I linked another rageshake that looks to be an instance of CreateSecretStorageDialog, we seem to decide that there is a key backup we should migrate the passphrase from (not entirely sure yet how and where that decision is made though) and then when we actually look for the megolm backup key, it doesn't seem to be there and throws an exception.

[jryans]

In my own testing, I was able to create a situation like this by doing:

1. Change setAccountData to throw an error
2. Bootstrap cross-signing for a fresh account
3. Sign out and back in so there's no locally cached cross-signing state, and skip verification during sign in
4. Try to bootstrap in Settings

In this path, we'll enter checkOwnCrossSigningTrust from bootstrap, which tries to get the master cross-signing private key from 4S account data, but it won't match the newer public key we successfully uploaded to the server, so we get a semi-cryptic error:

In such a situation, we need to reset cross-signing anyway so that our public and private keys are both the same "generation" and back in sync. If you unblock account data, resetting appears to work, so people shouldn't be stuck in such a situation, as they can still reset.

[jryans]

I linked another rageshake that looks to be an instance of CreateSecretStorageDialog, we seem to decide that there is a key backup we should migrate the passphrase from (not entirely sure yet how and where that decision is made though) and then when we actually look for the megolm backup key, it doesn't seem to be there and throws an exception.

I'll continue chasing this theory tomorrow.

[bwindels]

Right, that makes sense @jryans, and should be somewhat helped with the robustness work. I'm not sure why it would have ended up calling checkOwnCrossSigningTrust during the bootstrap, I'd expect you to end up in the first case because setupNewSecretStorage is true when triggered from a reset.

[bwindels]

There also seems to be another reset blocking issue that seems related to key backup, where you get this error:

2020-05-08T16:10:24.368Z I restoreWithCachedKey failed: Couldn't get key
Error: Couldn't get key
    at I.restoreKeyBackupWithCache (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:552106)
    at async hc._restoreWithCachedKey (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:4016156)
    at async hc._loadBackupStatus (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:4016575)

Probably best to break out in a separate issue.

[jryans]

I'm not sure why it would have ended up calling checkOwnCrossSigningTrust during the bootstrap, I'd expect you to end up in the first case because setupNewSecretStorage is true when triggered from a reset.

Ah, I meant that pressing the green bootstrap (so not a reset) descends down to checkOwnCrossSigningTrust and hits the error in the screenshot above: we have something in 4S already (it's just wrong generation of keys), so currently we go to the Cross-signing private keys found in secret storage branch and then into checkOwnCrossSigningTrust.

[jryans]

There also seems to be another reset blocking issue that seems related to key backup, where you get this error:

2020-05-08T16:10:24.368Z I restoreWithCachedKey failed: Couldn't get key
Error: Couldn't get key
    at I.restoreKeyBackupWithCache (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:552106)
    at async hc._restoreWithCachedKey (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:4016156)
    at async hc._loadBackupStatus (vector://vector/webapp/bundles/9f769ac7a648e2ec9507/vendors~init.js:2:4016575)

Probably best to break out in a separate issue.

Yes, let's break this out to a separate issue. I think that (somewhat confusing) log message mainly indicates we just don't have key backup key cached, so we'd need to read it from 4S instead.

[jryans]

Overall, I haven't been able to isolate actionable steps here, and my assumption is that @bwindels bootstrapping robustness will help with much / all of these issues.

For now, I'm returning this to the backlog, and we can re-test again once more robustness work has merged.

[richvdh]

This is still very much a thing; I just closed #26322 as a duplicate