Upgrade to Python 3.3 or newer

[steinn]

Mailpile is currently written in Python 2.7, which is no longer being actively developed and improved by the Python community. In order to stay current with the state of the art and reap the benefits of ongoing developments, Mailpile should upgrade to Python 3.3 (or newer).

(Rephrased by BRE for voting. Original issue text follows.)

I'm curious what Python version(s) Mailpile is targeting. It would be interesting to see Mailpile supporting Python 2.7 and 3.3+

The current code runs on Python 2.7 and it can probably run on Python 2.6 with minor changes (if any). By only supporting 2.7 and 3.3+ it should hopefully be fairly easy to use the same code base.

Python 2.6 end of life is in October, PEP 361, and therefor I think a good reason to not supporting it.
Python 2.7 end of life is in May 2015, PEP 373, which isn't far away!

Ubuntu is also planing on moving to Python 3 in the 14.04 Release, Ubuntu Python 3, which should be released according to schedule in April 2014. That alone is quite interesting and will hopefully get more people using Python 3.

An issue with updating the current code base to Python 3.3+ support is the Python GnuPGInterface library. It does not support Python 3.3 and probably never will (at least by upstream) since it has not been updated since 2002. Haven't looked into alternatives but there must be something.

If there is interest for his I'm willing to help.

[Ivoz]

Regarding the Python GnuPGInterface library, it seems there are newer versions (0.3.5) on both pypi and googlecode.

https://pypi.python.org/pypi/python-gnupg/
https://code.google.com/p/python-gnupg/

The newer PyPI tags claim compatibility all the way up to python 3.3, so it would be interesting to hear why the older sourceforge source is being used and not pypi.

[danx0r]

we may want to look at http://pythonhosted.org/six/

seems to facilitate writing 2.7-compatible code that is as future-proof against py3 changes as possible

[illume]

For best compatibility, single source is the way forward for python projects. Since the stated reason for staying with the legacy python 2.x was for best compatibility, it seems time to change the stance of the project to move to a single source.

It is almost April 2014 so more distros are moving to python 3. The python community is also moving in a big way to python 3.3/3.4. As previously said, python 2.7 is nearing end of life, meaning no security fixes, and no updates for platforms (new windows/osx etc). Writing a single source code base that works on 3.3+ and 2.6/2.7 isn't too much different, and lots of projects have documented their experiences with this method now.

As well as six, there is http://python-future.org/ and some of the projects with small compatibility modules include twisted, numpy, pygame, coverage, etc.

Here is an old blog post from 2009 showing how it is done: http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html

The jinja2 package has a _compat module which is about 100 lines long: https://github.com/mitsuhiko/jinja2/blob/master/jinja2/_compat.py

Since you are already using jinja2, I don't see why you couldn't vendorise its compatibility module :)

[BjarniRunar]

I agree that this is becoming more compelling all the time. We have already started to hit what at first glance looks like limitations in the python base libraries (e.g. #558 ) and it's unlikely that those bugs will ever get fixed on the 2.x branch.

I also did some very basic tests which implied that Python 3 is much better at releasing unused memory back to the operating system, and considering that RAM usage is one of our main bottlenecks that is a strong argument in favour of making the switch.

The core team (myself, @smari and @brennannovak) is discussing this.

[nifgraup]

Python 2.7 EOL has been moved 5 years into the future, see http://hg.python.org/peps/file/76d43e52d978/pep-0373.txt

[Ivoz]

Mailpile is still a relatively new application, so it shouldn't be aiming to stay behind.

[steinn]

@nifgraup yes but after 2015 there isn't a guarantee of regular releases, only if the core devs have interest in it. I think they are hoping that vendors (ubuntu, red hat, etc.) will handle releasing patches (this is my understanding from the python dev mailing list).

[Ivoz]

The releases will essentially just be bugfix releases, when python devs feel one is warranted for whatever comes up in the future. The promise is essentially saying they'll keep on officially paying attention to 2.7 and releasing bugfixes as needed until 2020.

[nidico]

This is also relevant due to STARTTLS support in imaplib starting in Python 3.2 (see #868).

[fawkesley]

+1 for Python3 support, especially as you're a "greenfield" project.

I'm trying to run on Python 3 - blocker so far is the python spambayes library.

[Ivoz]

For python 3 libraries, there is a port of PyDNS called Py3DNS. I'm unsure about its cross compatibility with Python 2. PGPDump is 2/3 compatible, same with MarkupSafe. Spambayes hasn't been updated since 2011 and I assume is python 2 compatible only, so you'd need to work out what to do about that. Is it the best python library for filtering spam?

[fawkesley]

I looked into porting spambayes a few weeks back but fell at the first hurdle - sourceforge!! Might be worth forking and bringing over to github, wonder how the code looks... :S

[konklone]

Note that since this ticket was filed, the communty is up to Python 3.4 (3.4.3, as of this comment).

[nklever]

One more argument to upgrade to Python 3.x is due to STARTTLS support in pop3lib starting in Python 3.4 (see POP3lib Doc)

[jonathan-s]

@BjarniRunar It seems like spambayes is the main blocker for python3, would something like https://github.com/dinever/antispam be acceptable?

[BjarniRunar]

@jonathan-s: Mailpile's antispam and statistical classifiers are written in a modular way, so dropping in a replacement for spambayes should be relatively straightforward. So yes, if it works well then that's a roadblock cleared! The bar isn't low - spambayes works really well in my experience - but replacing it with something slightly worse but actively being developed/improved is still an option, especially if it lets us move towards Python 3.x down the line.

Looking at the antispam package you linked, the API it currently offers appears too limited (in particular, I saw no way to ensure the classifier database was saved encrypted), but if it's under development it could certainly be fixed.

[jonathan-s]

@BjarniRunar I've been trying to find other packages related to bayesian spam. Sadly it seems like there are not many packages out there. And for the linked package it looks fairly simple and I'm unsure whether it'll add many more features (unless someone from mailpile community decides that these features are important).

For mailpile what features are important for spam classification? You mentioned that the classifier database needs to be saved encrypted.

[BjarniRunar]

@jonathan-s, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API.

Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification.

Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py

[jonathan-s]

Given the small footprint and the low-level implementation you're using it looks like it'll be pretty easy to copy the relevant files to use in mail pile internally.

On Sat 27. Jan 2018 at 15:13, Bjarni Rúnar Einarsson < ***@***.***> wrote: @jonathan-s <https://github.com/jonathan-s>, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API. Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification. Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBsEn7hd8O2V5TCta0XYd-4iXwyykZQks5tOy7qgaJpZM4A-GJ6> .

-- Sent from Gmail Mobile

[BjarniRunar]

@jonathan-s - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good...

[jonathan-s]

Taking over spambayes would be a daunting task. And the gains would be neglible. The downloads of the library seems to have shrunk by ~90% looking at the sourceforge statistics.

On Sat 27. Jan 2018 at 16:04, Bjarni Rúnar Einarsson < ***@***.***> wrote: @jonathan-s <https://github.com/jonathan-s> - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBsEmt7c0l_38A_OolS8GI69m8zVm2Lks5tOzrqgaJpZM4A-GJ6> .

-- Sent from Gmail Mobile

[intika]

💀 RIP Python 2 ^^

[darkuranium]

Any news on this? Python2's been EOL for nearly a year by now.

[bjmgeek]

It works for me on PyPy 3
Python 3.6.9 (7.3.2+dfsg-2, Sep 26 2020, 23:01:23)
[PyPy 7.3.2 with GCC 10.2.0]

The only thing was that requirements.txt included pydns but for python3 it needs to be py3dns

update. I thought it worked, but I didn't realize that it was using python2.7 explicitly, instead of just /usr/bin/env python. When my distro removed python2, it stopped working.

[BjarniRunar]

Thank you for reporting that, @bjmgeek ! We had done a bunch of prep work, but actually testing it never happened. It's really heartening to hear this, makes the whole thing seem much less daunting.

[bjmgeek]

It doesn't pass unit tests though. pydns being one sticking point.

[raboof]

It works for me on PyPy 3

on master? I tried running master with python3 but that immediately produced errors - I tried to stumble through fixing some of those in master...raboof:python3 (but be aware I don't actually know python ;) ) - is there some other trick I'm missing?

[reyman]

I'm also interested by an answer on this @BjarniRunar @bjmgeek ? There is only py3dns that blocks ?

[alerque]

Every time I setup a new server with webmail I come around to see if Mailpile has ditched Python 2 yet. I so want to love this project, but it needs a refresh.

[alerque]

I got excited there for a minute, what it seems like "closed as completed" actually means "won't fix in this project, but someday the multi-year stealth project might pick up where this left off". Color me not impressed yet since there doesn't seem to be a release cut or even on the near horizon, this was just a bulk closure of all issues.

[BjarniRunar]

Sorry to disappoint! The bulk closer wouldn't let me add a reason. Or I couldn't figure out how.

Thanks for taking the time to complain though, it's great to have a reminder as to why I'm mostly staying out of the public eye this time around! 👍

[alerque]

You have a project people (including myself) really want to be excited about, but you don't give them anything to hand onto, not even development status reports? Maybe if the dev work and even alpha releases / dev milestones towards a product were in view you might get some excitement and help building it.