move_file may cause data loss

[lilydjwg]

Days ago I found autojump forgot some of my frequent paths. After reading the code, I think this may be caused by that move_file isn't really atomic.

My system is Arch Linux and autojump version is v22.2.4.

On saving paths, autojump creates a temporary file, which by default, is in /tmp. /tmp isn't on the same filesystem with my autojump data file, so when move_file calls shutil.move, what really happens is copy and then delete. It's not atomic. Running with strace I can see there are multiple write calls to the data file. When multiple processes hit here, interweaving writes will result in a partial loss.

I suggest you use a temporary file in the same directory as the data file. I can provide a patch if you accept doing it this way.

[lilydjwg]

I've checked my backups. It shows about 11K data, more than 200 lines of history were lost in March 6 or 5. It's longer than what autojump keeps for as backup (only 24 hours) before I realize data loss had happened.

[sxe]

Hi lilydjwg,
i see data loss at my system as well. I make regular backups to the automjump.txt file to be able to restore it but it's happening so often now that it get really annoying. At first I thought there is maybe some kind of database cleanup going on that i did not figure out so far.

[lilydjwg]

@sxe I've forked and made some change to avoid the situation I've described. Would you try that and see if it solves the problem for you?

[sxe]

I will and report back, thx! Give me a couple of days to get some data.

[sxe]

@lilydjwg looks really good so far.
Down below is a log of my backup files.
The <- indicates a day where i had data loss. I copied the backup file after it, that's the reason the file size gets bigger again (first column).
As you can see after applying your patch it did not happen again.
At the very least it got much better but i think it's fixed.
Thanks a lot.

6469 14. Feb 18:07 autojump.txt.150214
6672 15. Feb 14:13 autojump.txt.150215
6672 16. Feb 20:24 autojump.txt.150216
4565 17. Feb 16:42 autojump.txt.150217 <-
4586 18. Feb 20:09 autojump.txt.150218
4541 19. Feb 21:36 autojump.txt.150219 <-
4351 20. Feb 18:12 autojump.txt.150220
4621 21. Feb 22:04 autojump.txt.150221
6128 22. Feb 18:54 autojump.txt.150222
5159 23. Feb 19:12 autojump.txt.150223 <-
5144 24. Feb 20:32 autojump.txt.150224 <-
6416 25. Feb 21:24 autojump.txt.150225
6862 26. Feb 21:17 autojump.txt.150226
7527 27. Feb 22:58 autojump.txt.150227
5868 28. Feb 17:10 autojump.txt.150228 <-
4812 1. Mär 12:21 autojump.txt.150301 <-
4712 2. Mär 20:57 autojump.txt.150302 <-
4958 3. Mär 20:43 autojump.txt.150303
111 4. Mär 21:54 autojump.txt.150304 <-
853 5. Mär 13:49 autojump.txt.150305
3248 6. Mär 18:27 autojump.txt.150306
8059 7. Mär 21:06 autojump.txt.150307
8070 8. Mär 13:24 autojump.txt.150308
7093 9. Mär 13:15 autojump.txt.150309 <-
5998 10. Mär 20:43 autojump.txt.150310 <-
5390 11. Mär 23:32 autojump.txt.150311 <-
664 12. Mär 22:39 autojump.txt.150312 <-
8252 13. Mär 21:36 autojump.txt.150313 -- startet using your build
10162 14. Mär 21:37 autojump.txt.150314
10443 15. Mär 15:47 autojump.txt.150315
16860 16. Mär 14:17 autojump.txt.150316
16860 16. Mär 14:17 autojump.txt

[AaronM04]

I have also experienced data loss every week or so.