-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[8.2] Test plan encrypt all and decrypt all #6
Comments
Let me add some points:
@SergioBertolinSG adds:
|
Btw, known limitation. We only decrypt the payload (files in data/user/files). files in trashbin and versions are not decrypted but this shouldn't be an issue as long as the admin keep the encryption modules enabled. Alternatively he could use the occ script to clean up the trashbins and versions |
At the end of the encryption when listing the users with passwords, ldap users show the uid, not the username. I think it could be a bit confusing.
|
@schiesbn How can I decrypt only for one user? |
When encrypting everything only admin user can login. It says owncloud is in a single user mode. Opened issue here owncloud/core#19213 |
I think the problem is that LDAP users can even have multiple login names, so I don't really know what to show here... @blizzz any idea what we can/should show instead for ldap users? We could show the display name instead of the uid. But the display name is not guaranteed to be unique. |
you can call decrypt-all with a user-id as parameter, see:
(same "problem" as above, for LDAP users you need the ID) |
After decrypting only one user's files, it has disabled encryption app.
Also when asking for login password, it is a single user but it writes 'users' in plural.
|
It's not plural, it's missing the apostrophe in user's |
That's the intended behavior. The idea is that if you want to decrypt all you also want to disable encryption, see the description:
The main purpose of adding a specific user as parameter is to allow to disable encryption for small instances where you don't have a recovery key but know the users login password or can ask them to enter it. Maybe there is one use case in a enterprise setup, if the decryption failed at some point and you want to call it again for a specific user. But every time it is coupled with the assumption that you want to disable encryption. |
@cmonteroluque thanks, typo is fixed here owncloud/core#19223 |
@schiesbn ok, thanks. After decrypting files for only one user, enabling again encryption and running encrypt-all it returns errors related with locked files: is it a problem? or it a bad use of encrypt-all ?
|
@SergioBertolinSG I can reproduce it and will look into it. This worked in the past, maybe the way locking works changed recently?
also this file is really confusing. Where does it come from? The encrypt all script doesn't create such files... @PVince81 @icewind1991 any idea? Is this something locking specific? |
@schiesbn this is just a hash of the file path, not the real path to the file. |
@SergioBertolinSG please specify the version you are testing, a few days ago a locking issue was fixed on master. |
Version is beta1 {"installed":true,"maintenance":false,"version":"8.2.0.6","versionstring":"8.2 beta1","edition":"Enterprise"} |
I could re-produce it. The copy operation throws an exception (https://github.com/owncloud/core/blob/master/lib/private/files/view.php#L788), so we never release the lock. Everywhere where we create locks we should have a try-catch block to remove the lock correctly if a exception happens. (I will rise a separate issue for it). Beside that I will try to find out why the exception happens. |
Issue regarding the locking problem: owncloud/core#19242 |
OK, throwing a exception on copy is OK in this case. We try to copy a already encrypted file and because we don't have the user password we don't have his private key -> operation will fail with a exception. This is catched and handled correctly in the encryptAll() script. We just need to make sure that locking frees the file correctly. |
@SergioBertolinSG this pr should fix the issue owncloud/core#19247 |
What is probably supposed to be shown is the owncloud username (as displayed in the very first column on the users page), to clearly identify a user. If however the used login name is being displayed, then it does not identify a user. Could you output the userid instead, @schiesbn? |
@schiesbn While I find the occ encryption:encrypt-all interface nice for few users few files, I think it is not comfortable for massive set ups.
To run the encryption all the night and see the results next day. Perhaps this suggestion is a huge improvement. What do you think? |
@SergioBertolinSG he could also run it in a screen session and look at the output at the next day. As you said we need some interaction, so I don't see how this should work. Also the progress bar was a feature people asked all the time in the past so that they get a feeling how long it will take and see that something happens. |
@blizzz The userid is what we display at the moment, see #6 (comment) @SergioBertolinSG thought that this could be confusing. But I don't know what a typical LDAP admin would expect. As we also use it in the user management I think this is the right ID. |
@SergioBertolinSG please, related to encryption, take into account as well: owncloud/core#18482 thx |
@schiesbn @DeepDiver1975 Encrypt-all command having 1 million of ldap users takes too much time (many days). |
@SergioBertolinSG do all 1 million people have files on ownCloud? How many files? What is the size of the files? Would be good to have some additional numbers. Maybe you can also evaluate the different steps. First we create a key pair for each user, how long does this take and when does it starts to encrypt the files? Generally speaking: It is expensive to encrypt all files for all users. That's why in general "encryption yes/no" should always be a decision by the admin at the beginning. This scripts are only there is someone really, really wants to switch a existing installation and can't live with the default behavior (files get encrypted the next time you touch it). It is nothing which we want the admins to turn it on and off all the time. I don't see how this can be speed up substantially. We need to read every file, encrypt it and write it again. Also 1 million sounds quite academic to me. I think even the largest installations are far away from 1 million users. But let's look at the numbers and see if we can improve at least some parts. (But I fear that 1 million users, all have a few MB or even GB of data will always take looooong) |
It takes too much in creating all the key-pair. Perhaps 1 million is too much, but I don't think encrypting this way is suitable for even 50000 users with very few files for example. |
After 24h running it has created ~29000 key-pair. |
As said, I fear that we can't do much about it. To illustrate it I wrote this small test script:
It just perform the basic openSSL operations to create the key pair and to encrypt it. This script takes something between 0.055 ans 0.24 seconds if I run it multiple times. For now let's assume that it takes 0.1 seconds in average for every keypair (which is probably to optimistic, but anyway). For 1 million users this means 1666 minutes ~ 27h (worst case (0.24s/key): 4000 minutes ~ 66h) There is nothing we can do. That's just openssl. In our case we need to write the keys to the hard disc and update the file cache (maybe something we could defer). But we would always have 50 minutes + writing 100.000 files to the disc (private/public key for 50.000 users). |
Just to get a idea how much time we could save by disabling the file cache update for each key. Can you try to edit lib/private/encryption/keys/storage.php and add here https://github.com/owncloud/core/blob/master/lib/private/encryption/keys/storage.php#L64 this line:
And then run the test again with 50.000 user. This way all keys should be created in about 200 minutes (worst case from above) + the time needed to write the keys to the hard disc, so the absolute minimum if the file cache is disabled. This is just a dirty hack to disable the cache and see if it would help at all. That's the only thing I see which could be improved. |
Regarding the deep folder structure I found this owncloud/core#19309 |
@schiesbn I've retested #6 (comment) with last master (2015-09-23) and I an facing the same locking errors. Shall we reopen thttps://github.com/owncloud/core/issues/19242 ? |
Using decrypt-all without a parameter it doesn't fail hard. This is a bug.
|
Trying to encrypt all files when some users have the recovery key enabled it has raised the locking error as well.
|
@schiesbn this test "run with multiple users some has encrypted files and some has already decrypted files" means that decrypt-all should work also for all users without putting a parameter as username? |
Using encrypt-all when key-pair where created already returns an output not prepared for this case owncloud/core#19332 |
Yes, in this case the script should just decrypt the files from the remaining users so that at the end the files for all users are decrypted |
Running the decryption to all users while some has recovery key enabled and some don't using patch from owncloud/core#19340) oC keeps in single-user mode. Besides that and disabling single-user mode manually it works correctly. Only users with recovery key has their files decrypted while the rest remain encrypted. |
did you saw any error message in your terminal or apache/owncloud log? From the code path I don't see how this could happen. Either you reach the end and we disable single user mode or we disable it in the catch-block |
I can't re-produce it. Can you explain the step to reproduce it (please keep the example as small as possible, as few users, files and settings as possible to isolate the problem) |
No errors in the terminal.
|
OK, the problem happens only when decrypting-all and later encrypting-all again. So the problem is the same as with #6 (comment) |
still can't re-produce it. Works as expected on my system (master d7a923671fa4b10f74dc615b583f6d4f5935d45b) |
hm, looks OK. The warnings shouldn't be a problem, though I will add a if statement to avoid the warning. Still don't see how you can neither reach the end nor the catch block to execute one of the two
|
8.2 introduces occ commands to encrypt and decrypt all files for all users.
this has to be tested and verified with
@SergioBertolinSG adds:
[ 🕓 ] Encrypt all having only 1 user with 100.000 files of 1KiB. (It takes long time, perhaps three days or so which leads us again to owncloud/core#19358 )
The text was updated successfully, but these errors were encountered: