Mailpile cannot find message in Maildir

[ghost]

TL-DR: Mailpile seems to index messages in Maildirs correctly, but cannot find them again.

---

I have set up Mailpile with a Maildir. Searching and viewing the Inbox etc. works fine. However, when I want to view a message (either on the command line or the web client), I get the following exception:

Traceback (most recent call last): File "/home/name/Downloads/mailpile/Mailpile/mailpile/commands.py", line 186, in _run return self._finishing(command, command(self, *args, **kwargs)) File "/home/name/Downloads/mailpile/Mailpile/mailpile/commands.py", line 174, in command return self.command(*args, **kwargs) File "/home/name/Downloads/mailpile/Mailpile/mailpile/plugins/search.py", line 307, in command expand=[email])) File "/home/name/Downloads/mailpile/Mailpile/mailpile/plugins/search.py", line 147, in __init__ result['message'] = self._message_details([exp_email])[0] File "/home/name/Downloads/mailpile/Mailpile/mailpile/plugins/search.py", line 68, in _message_details tree = email.get_message_tree() File "/home/name/Downloads/mailpile/Mailpile/mailpile/mailutils.py", line 816, in get_message_tree msg = self.get_msg() File "/home/name/Downloads/mailpile/Mailpile/mailpile/mailutils.py", line 696, in get_msg fd = self.get_file() File "/home/name/Downloads/mailpile/Mailpile/mailpile/mailutils.py", line 692, in get_file return self.get_mbox_ptr_and_fd()[2] File "/home/name/Downloads/mailpile/Mailpile/mailpile/mailutils.py", line 683, in get_mbox_ptr_and_fd fd = mbox.get_file_by_ptr(msg_ptr) File "/home/name/Downloads/mailpile/Mailpile/mailpile/mailutils.py", line 313, in get_file_by_ptr return self.get_file(msg_ptr[3:]) File "/usr/lib/python2.7/mailbox.py", line 372, in get_file f = open(os.path.join(self._path, self._lookup(key)), 'rb') File "/usr/lib/python2.7/mailbox.py", line 542, in _lookup raise KeyError('No message with key: %s' % key) KeyError: u'No message with key: 1375542072_0.2742.hostname

(name and hostname replaced for privacy)

In the corresponding dir, the filename is 1375542072_0.2742.hostname,U=4283,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,RS.

Opening the mailbox with the standard Python mailbox.Maildir class gives a list of keys with the stuff after the comma included. I couldn't figure out however, where this gets lost in Mailpile.

[ghost]

Getting to understand this problem better. it seems as if offlineimap adds comma-separated fields (fmd5, etc.) before the colon in the maildir filename, leading into problems with Mailpile's comma-based serialization of filenames.

[BjarniRunar]

Good catch, thanks! I will try and work on a fix today.

[ghost]

Awesome - however, it doesn't work as of now.

Line 309 in mailutils.py caught my eye:
return '%s%s' % (idx_id, toc_id.split(',')[0].split(':')[0])

The filenames which offlineimap creates have the the format.

uniqid,fmd5=blablabla,etc=etc:f=r,a=b

where everything before the colon is part of the maildir id.

Thus, you would get the id by only doing:

return '%s%s' % (idx_id, toc_id.split(':')[0])

In addition, there are (probably) two other bugs:
a) I believe that the problem still persists that an array of ids (which, as of above my contain commas) gets serialized by ','.join (for example in search.py:483, msg_info[self.MSG_PTRS] = ','.join(msg_ptrs)) and consequently get unserialized incorrectly, because of the commas within the ids.
b) on Windows, offlineimap uses ! instead of : to separate the flags from the id, so that case should also be considered.

[BjarniRunar]

Again, very useful information.

Could you test 448ab77 ? That adds both a Windows-specific Maildir class using ! instead of : and it takes care to quote msg_ptr values so they never contain characters that would break the metadata index.

[ghost]

Thanks so much! Loading messages works now perfectly (search seems to be broken, but I suspect this is an unrelated issue with URL mappings).

I don't have a Windows environment at the moment to test, so I can't confirm that. But I will close the bug anyway.

Thanks again! Great work!