Sendmail introduced a
new API beginning with version 8.10 -
libmilter. The milter module for Python
provides a python interface to libmilter that exploits all its features.
Sendmail 8.12 officially releases libmilter. Version 8.12 seems to be more robust, and includes new privilege separation features to enhance security. Even better, sendmail 8.13 supports socket maps, which makes pysrs much more efficient and secure. I recommend upgrading.
Release 0.8.0 is the first Sourceforge release. It supports Python-2.4, and provides an option to accept mail that gets an SPF softfail or fails the 3 strikes rule, provided the alleged sender accepts a DSN explaining the problem. Python-2.3 is no longer supported by the reworked mime.py module, although API changes could be backported. There are too many incompatible changes to the python email package.
Release 0.7.2 tightens the authentication screws with a "3 strikes and you're out" policy. A sender must have a valid PTR, HELO, or SPF record to send email. Specific senders can be whitelisted using the "delegate" option in the spf configuration section by adding a default SPF record for them. The PTR and HELO are required by RFC anyway, so this is not an unreasonable requirement. There is now a coherent policy for an SPF softfail result. A softfail is accepted if there is a valid PTR or HELO, or if the domain is listed in the "accept_softfail" option of the spf configuration section. A neutral result is accepted by default if there is a valid PTR or HELO, (and the SPF record was not guessed), unless the domain is listed in the "reject_neutral" option. Common forms of PTR records for dynamic IPs are recognized, and do not count as a valid PTR. This does not prevent anyone from sending mail from a dynamic IP - they just need to configure a valid HELO name or publish an SPF record.
As SPF adoption continues to rise, forged spam is not getting through. So
spammers are publishing their SPF records as predicted. The 0.7.2 RPM
now provides the rhsbl sendmail hack so that spammer domains
can be blacklisted. With the RPM installed, add a line like the following
to your sendmail.mc.
HACK(rhsbl,`blackholes.example.com',"550 Rejected: " $&{RHS} " has been spamming our customers.")dnl
Of course, spammers are now starting to register throwaway domains. The next thing we need is a custom DNS server, in Python, that can recognize patterns. For instance, one spammer registers ded304.com, ded305.com, ded306.com, etc. We also need the custom DNS server to let SPF classic clients check SES (which will be part of pysrs). The Twisted Python framework provides a custom DNS server - but I would like a smaller implementation for our use.
The RPM for release 0.7.0 moves the config file and socket locations to /etc/mail and /var/run/milter respectively. We now parse Microsoft CID records - but only hotmail.com uses them. They seem to have applied for a patent on the brilliant idea of examining the mail headers to see who the message is from. We aren't doing that here, so not to worry - but I am not a lawyer, so if you are worried, change spf.py around line 626 to return None instead of calling CIDParser(). There is a new option to reject mail with no PTR and no SPF.
Microsoft is pushing an anti-opensource license for their pending patent along with their sender-ID proposal before the IETF. It is royalty free - but requires anyone distributing a binary they've compiled from source to sign a license agreement. The Apache Software Foundation explains the problem with sender-ID, and Debian concurs. Since the Microsoft license is incompatible with free software in general and the GPL in particular, Python milter will not be able to implement sender-ID in its current form. This was, no doubt, Microsoft's intent all along.
Sender-ID attempts to do for RFC2822 headers what SPF does for RFC2821 headers. Unlike SPF, it has never been tried, and is encumbered by a stupid patent. I recommend ignoring it and continuing to implement and improve SPF until a working and unencumbered proposal for RFC2822 headers surfaces.
Release 0.6.6 adds support for SPF,
a protocol to prevent forging of the envelope from address.
SPF support requires pydns.
The included spf.py module is an updated version of the original 1.6
version at wayforward.net.
The updated version tracks the draft RFC and test suite.
The FAQ addresses how to get started with SPF.
Release 0.6.1 adds a full milter based dspam application.
I have selected the dspam bayes filter project and packaged it for python. Release 0.6.0 offers a simple application of dspam I call "header triage", which rejects messages with spammy headers. To use header triage, you must have DSPAM installed, and select a dictionary that is well moderated by someone who gets lots of spam. That dictionary can be used to block spam that is obvious from the headers (e.g. X-Mailer and Subject) before it ties up any more resources. I have yet to see any false positives from this approach (check the milter log), but if there are, the sender will get a REJECT with the message "Your message looks spammy."
To Bcc a message, call self.add_recipient(rcpt) in envfrom after
determining whether you want to copy (e.g. whether the sender is local). For
example,
def envfrom(...
...
if len(t) == 2:
self.rejectvirus = t[1] in reject_virus_from
if t[0] in wiretap_users.get(t[1],()):
self.add_recipient(wiretap_dest)
if t[1] == 'mydomain.com':
self.add_recipient('<copy-%s>' % t[0])
...
To make this a generic feature requires thinking about how the configuration would look. Feel free to make specific suggestions about config file entries. Be sure to handle both Bcc and file copies, and designating what mail should be copied. How should "outgoing" be defined? Implementing it is easy once the configuration is designed.
At the lowest level, the 'milter' module provides a thin wrapper around the sendmail libmilter API. This API lets you register callbacks for a number of events in the process of sendmail receiving a message via SMTP. These events include the initial connection from a MTA, the envelope sender and recipients, the top level mail headers, and the message body. There are options to mangle all of these components of the message as it passes through the milter.
At the next level, the 'Milter' module (note the case difference) provides a Python friendly object oriented wrapper for the low level API. To use the Milter module, an application registers a 'factory' to create an object for each connection from a MTA to sendmail. These connection objects must provide methods corresponding to the libmilter callback events.
Each event method returns a code to tell sendmail whether to proceed with processing the message. This is a big advantage of milters over other mail filtering systems. Unwanted mail can be stopped in its tracks at the earliest possible point.
The Milter.Milter class provides default implementations for event methods that do nothing, and also provides wrappers for the libmilter methods to mutate the message.
The 'spf' module provides an implementation of SPF useful for detecting email forgery.
The 'mime' module provides a wrapper for the Python email package that fixes some bugs, and simplifies modifying selected parts of a MIME message.
Finally, the bms.py application is both a sample of how to use the Milter and spf modules, and the beginnings of a general purpose SPAM filtering, wiretapping, SPF checking, and Win32 virus protecting milter. It can make use of the pysrs package when available for SRS/SES checking and the pydspam package for Bayesian content filtering. SPF checking requires pydns. Configuration documentation is currently included as comments in the sample config file for the bms.py milter.
Python milter is under GPL. The authors can probably be convinced to change this to LGPL if needed.
The Python milter package includes a sample milter that replaces dangerous
attachments with a warning message, discards mail addressed to
MAILER-DAEMON, and demonstrates several SPAM abatement strategies.
The MimeMessage class to do this used to be based on the
mimetools and multifile standard python packages.
As of milter version 0.6.0, it is based on the email standard
python packages, which were derived from the
mimelib project.
The MimeMessage class patches several bugs in the email package,
and provides some backward compatibility.
The "defang" function of the sample milter was inspired by MIMEDefang, a Perl milter with flexible attachment processing options. The latest version of MIMEDefang uses an apache style process pool to avoid reloading the Perl interpreter for each message. This makes it fast enough for production without using Perl threading.
mailchecker is a Python project to provide flexible attachment processing for mail. I will be looking at plugging mailchecker into a milter.
TMDA is a Python project to require confirmation the first time someone tries to send to your mailbox. This would be a nice feature to have in a milter.
There is also a Milter community website where milter software and gory details of the API are discussed.
For example, the HTML parsing feature to remove scripts from HTML attachments is rather CPU intensive in pure python. Using the C replacement for sgmllib greatly speeds things up.
| Operating System | Compiler | Python | Sendmail | milter |
|---|---|---|---|---|
| Mandrake 8.0 | gcc-3.0.1 | 2.1.1 | 8.12.0 | 0.3.3 |
| Mandrake 8.0 | gcc-2.96 | 2.0 | 8.11.2 | 0.3.6 |
| RedHat 6.2 | egcs-1.1.2 | 2.2.2 | 8.11.6 | 0.5.4 |
| RedHat 7.1 | gcc-2.96 | ? | 8.12.1 | 0.3.5 |
| RedHat 7.3 | gcc-2.96 | 2.2.2 | 8.11.6 | 0.5.5 |
| RedHat 7.3 | gcc-2.96 | 2.3.3 | 8.13.1 | 0.7.2 |
| RedHat 8.0 | gcc-3.2 | 2.2.1 | 8.12.6 | 0.5.2 |
| Debian Linux | gcc-2.95.2 | 2.1.1 | 8.12.0 | 0.3.7 |
| Debian Linux | gcc-3.2.2 | 2.2.2 | 8.12.7 | 0.5.4 |
| AIX-4.1.5 | gcc-2.95.2 | 2.1.1 | 8.11.5 | 0.3.3 |
| AIX-4.1.5 | gcc-2.95.2 | 2.1.1 | 8.12.1 | 0.3.4 |
| AIX-4.1.5 | gcc-2.95.2 | 2.1.3 | 8.12.3 | 0.4.2 |
| AIX-4.1.5 | gcc-2.95.2 | 2.2.3 | 8.13.1 | 0.7.1 |
| Slackware 7.1 | ? | ? | 8.12.1 | 0.3.8 |
| Slackware 9.0 | gcc-3.2.2 | 2.2.3 | 8.12.9 | 0.5.4 |
| OpenBSD | ? | 2.3.3? | 8.13.1? | 0.7.2 |
| SuSE 7.3 | gcc-2.95.3 | 2.1.1 | 8.12.2 | 0.3.9 |
| FreeBSD | gcc-2.95.3 | 2.2.1 | 8.12.3 | 0.4.0 |
| FreeBSD | gcc-2.95.3 | 2.2.2 | ? | 0.5.5 |
| FreeBSD 4.4 | gcc-2.95.3 | ? | 8.12.10 | 0.6.6 |
Sendmail-8.12 renames libsmutil.a to libsm.a. Unfortunately, libsm.a is an important AIX system shared library. Therefore, I rename libsm.a back to libsmutil.a for AIX. This presents a problem for setup.py.
/usr/include/libmilter/mfapi.h.
If you do modify the SRPM, I suggest renaming libsmutil.a
to libsm.a - just like sendmail-8.12 will. If you manually install
mfapi.h or don't rename libsmutil.a, you'll
need to force libs = ["milter", "smutil"] in setup.py.
If you have installed python2, and want
python-milter to use python2, add python=python2 to setup.cfg
and build with python2 setup.py bdist_rpm.
If y'all trust me, you can pick up source and binary sendmail RPMs for RH6.2 from my linux downloads directory. The lastest RPMs were built by taking a RH7.2 SRPMS and removing some RPM features from the spec file that RH6.2 doesn't support, then recompiling on RH6.2. You can check this by installing the RH7.2 SRPM, then diffing my sendmail.spec with theirs. Then run "rpm -bb sendmail-rhmilter.spec" when you are satisfied.
If you have installed python2, and want
python-milter to use python2, add python=python2 to setup.cfg
and build with python2 setup.py bdist_rpm.
You'll need to install the sendmail-devel package to compile milter.