Viewable With Any Browser Your vote? I Disagree I Agree

Sendmail Milters in Python

by Jim Niemira and Stuart D. Gathman
This web page is written by Stuart D. Gathman
and
sponsored by Business Management Systems, Inc.
Last updated Jun 09, 2005

See the FAQ | Download now | Subscribe to mailing list | Overview | pydspam | libdspam

A Python Sendmail introduced a new API beginning with version 8.10 - libmilter. The milter module for Python provides a python interface to libmilter that exploits all its features.

Sendmail 8.12 officially releases libmilter. Version 8.12 seems to be more robust, and includes new privilege separation features to enhance security. Even better, sendmail 8.13 supports socket maps, which makes pysrs much more efficient and secure. I recommend upgrading.

Recent Changes

Python milter is being moved to pymilter Sourceforge project for development.

Release 0.8.0 is the first Sourceforge release. It supports Python-2.4, and provides an option to accept mail that gets an SPF softfail or fails the 3 strikes rule, provided the alleged sender accepts a DSN explaining the problem. Python-2.3 is no longer supported by the reworked mime.py module, although API changes could be backported. There are too many incompatible changes to the python email package.

Release 0.7.2 tightens the authentication screws with a "3 strikes and you're out" policy. A sender must have a valid PTR, HELO, or SPF record to send email. Specific senders can be whitelisted using the "delegate" option in the spf configuration section by adding a default SPF record for them. The PTR and HELO are required by RFC anyway, so this is not an unreasonable requirement. There is now a coherent policy for an SPF softfail result. A softfail is accepted if there is a valid PTR or HELO, or if the domain is listed in the "accept_softfail" option of the spf configuration section. A neutral result is accepted by default if there is a valid PTR or HELO, (and the SPF record was not guessed), unless the domain is listed in the "reject_neutral" option. Common forms of PTR records for dynamic IPs are recognized, and do not count as a valid PTR. This does not prevent anyone from sending mail from a dynamic IP - they just need to configure a valid HELO name or publish an SPF record.

As SPF adoption continues to rise, forged spam is not getting through. So spammers are publishing their SPF records as predicted. The 0.7.2 RPM now provides the rhsbl sendmail hack so that spammer domains can be blacklisted. With the RPM installed, add a line like the following to your sendmail.mc.

HACK(rhsbl,`blackholes.example.com',"550 Rejected: " $&{RHS} " has been spamming our customers.")dnl

Of course, spammers are now starting to register throwaway domains. The next thing we need is a custom DNS server, in Python, that can recognize patterns. For instance, one spammer registers ded304.com, ded305.com, ded306.com, etc. We also need the custom DNS server to let SPF classic clients check SES (which will be part of pysrs). The Twisted Python framework provides a custom DNS server - but I would like a smaller implementation for our use.

The RPM for release 0.7.0 moves the config file and socket locations to /etc/mail and /var/run/milter respectively. We now parse Microsoft CID records - but only hotmail.com uses them. They seem to have applied for a patent on the brilliant idea of examining the mail headers to see who the message is from. We aren't doing that here, so not to worry - but I am not a lawyer, so if you are worried, change spf.py around line 626 to return None instead of calling CIDParser(). There is a new option to reject mail with no PTR and no SPF.

Microsoft is pushing an anti-opensource license for their pending patent along with their sender-ID proposal before the IETF. It is royalty free - but requires anyone distributing a binary they've compiled from source to sign a license agreement. The Apache Software Foundation explains the problem with sender-ID, and Debian concurs. Since the Microsoft license is incompatible with free software in general and the GPL in particular, Python milter will not be able to implement sender-ID in its current form. This was, no doubt, Microsoft's intent all along.

Sender-ID attempts to do for RFC2822 headers what SPF does for RFC2821 headers. Unlike SPF, it has never been tried, and is encumbered by a stupid patent. I recommend ignoring it and continuing to implement and improve SPF until a working and unencumbered proposal for RFC2822 headers surfaces.

SPF logo Release 0.6.6 adds support for SPF, a protocol to prevent forging of the envelope from address. SPF support requires pydns. The included spf.py module is an updated version of the original 1.6 version at wayforward.net. The updated version tracks the draft RFC and test suite.

The FAQ addresses how to get started with SPF.

Release 0.6.1 adds a full milter based dspam application.

I have selected the dspam bayes filter project and packaged it for python. Release 0.6.0 offers a simple application of dspam I call "header triage", which rejects messages with spammy headers. To use header triage, you must have DSPAM installed, and select a dictionary that is well moderated by someone who gets lots of spam. That dictionary can be used to block spam that is obvious from the headers (e.g. X-Mailer and Subject) before it ties up any more resources. I have yet to see any false positives from this approach (check the milter log), but if there are, the sender will get a REJECT with the message "Your message looks spammy."

Enough Already!

Nearly a dozen people have emailed me begging for a feature to copy outgoing and/or incoming mail to a backup directory by user. Ok, it looks like this is a most requested feature for 0.5.6. In the meantime, here are some things to consider:

To Bcc a message, call self.add_recipient(rcpt) in envfrom after determining whether you want to copy (e.g. whether the sender is local). For example,

  def envfrom(...
    ...
    if len(t) == 2:
      self.rejectvirus = t[1] in reject_virus_from
      if t[0] in wiretap_users.get(t[1],()):
	self.add_recipient(wiretap_dest)
      if t[1] == 'mydomain.com':
        self.add_recipient('<copy-%s>' % t[0])
      ...

To make this a generic feature requires thinking about how the configuration would look. Feel free to make specific suggestions about config file entries. Be sure to handle both Bcc and file copies, and designating what mail should be copied. How should "outgoing" be defined? Implementing it is easy once the configuration is designed.

Overview

This package provides a robust toolkit for Python milters, and the beginnings of a general purpose mail filtering system written in Python.

At the lowest level, the 'milter' module provides a thin wrapper around the sendmail libmilter API. This API lets you register callbacks for a number of events in the process of sendmail receiving a message via SMTP. These events include the initial connection from a MTA, the envelope sender and recipients, the top level mail headers, and the message body. There are options to mangle all of these components of the message as it passes through the milter.

At the next level, the 'Milter' module (note the case difference) provides a Python friendly object oriented wrapper for the low level API. To use the Milter module, an application registers a 'factory' to create an object for each connection from a MTA to sendmail. These connection objects must provide methods corresponding to the libmilter callback events.

Each event method returns a code to tell sendmail whether to proceed with processing the message. This is a big advantage of milters over other mail filtering systems. Unwanted mail can be stopped in its tracks at the earliest possible point.

The Milter.Milter class provides default implementations for event methods that do nothing, and also provides wrappers for the libmilter methods to mutate the message.

The 'spf' module provides an implementation of SPF useful for detecting email forgery.

The 'mime' module provides a wrapper for the Python email package that fixes some bugs, and simplifies modifying selected parts of a MIME message.

Finally, the bms.py application is both a sample of how to use the Milter and spf modules, and the beginnings of a general purpose SPAM filtering, wiretapping, SPF checking, and Win32 virus protecting milter. It can make use of the pysrs package when available for SRS/SES checking and the pydspam package for Bayesian content filtering. SPF checking requires pydns. Configuration documentation is currently included as comments in the sample config file for the bms.py milter.

Python milter is under GPL. The authors can probably be convinced to change this to LGPL if needed.

What is a milter?

Milters can run on the same machine as sendmail, or another machine. The milter can even run with a different operating system or processor than sendmail. Sendmail talks to the milter via a local or internet socket. Sendmail keeps the milter informed of events as it processes a mail connection. At any point, the milter can cut the conversation short by telling sendmail to ACCEPT, REJECT, or DISCARD the message. After receiving a complete message from sendmail, the milter can again REJECT or DISCARD it, but it can also ACCEPT it with changes to the headers or body.

What can you do with a milter?

  • A milter can DISCARD or REJECT spam based based on algorithms scripted in python rather than sendmail's cryptic "cf" language.
  • A milter can alter or remove attachments from mail that are poisonous to Windows.
  • A milter can scan for viruses and clean them when detected.
  • A milter scans outgoing as well as incoming mail.
  • A milter can add and delete recipients to forward or secretly copy mail.
  • For more ideas, check the Milter Web Page.
  • Documentation for the C API is provided with sendmail. Miltermodule provides a thin python wrapper for the C API. Milter.py provides a simple OO wrapper on top of that.

    The Python milter package includes a sample milter that replaces dangerous attachments with a warning message, discards mail addressed to MAILER-DAEMON, and demonstrates several SPAM abatement strategies. The MimeMessage class to do this used to be based on the mimetools and multifile standard python packages. As of milter version 0.6.0, it is based on the email standard python packages, which were derived from the mimelib project. The MimeMessage class patches several bugs in the email package, and provides some backward compatibility.

    The "defang" function of the sample milter was inspired by MIMEDefang, a Perl milter with flexible attachment processing options. The latest version of MIMEDefang uses an apache style process pool to avoid reloading the Perl interpreter for each message. This makes it fast enough for production without using Perl threading.

    mailchecker is a Python project to provide flexible attachment processing for mail. I will be looking at plugging mailchecker into a milter.

    TMDA is a Python project to require confirmation the first time someone tries to send to your mailbox. This would be a nice feature to have in a milter.

    There is also a Milter community website where milter software and gory details of the API are discussed.

    Is a milter written in python efficient?

    The python milter process is multi-threaded and startup cost is incurred only once. This is much more efficient than some implementations that start a new interpreter for each connection. Testing in a production environment did not use a significant percentage of the CPU. Furthermore, python is easily extended in C for any step requiring expensive CPU processing.

    For example, the HTML parsing feature to remove scripts from HTML attachments is rather CPU intensive in pure python. Using the C replacement for sgmllib greatly speeds things up.

    Goals

  • Implement RRS - a backdoor for non-SRS forwarders. User lists non-SRS forwarder accounts (perhaps in ~/.forwarders), and a util provides a special local alias for the user to give to the forwarder. Alias only works for mail from that forwarder. Milter gets forwarder domain from alias and uses it to SPF check forwarder. Requires milter to have read access to ~/.forwarders or else a way for user to submit entries to milter database.
  • The bms.py milter has too many features. Create a framework where numerous small feature modules can be plugged together in the configuration.
  • Create a pure python substitute for miltermodule and libmilter that implements the libmilter protocol in python.
  • Find or write a faster implementation of sgmllib. The sgmlop package is not very compatible with Python-2.1 sgmllib, but it is a start, and is supported in milter-0.4.5 or later.
  • Implement all or most of the features of MIMEDefang.
  • Follow the official Python coding standards more closely.
  • Make unit test code more like other python modules.
  • Confirmed Installations

    Please email me if you successfully install milter on a system not mentioned below.

    Operating System Compiler Python Sendmail milter
    Mandrake 8.0gcc-3.0.12.1.18.12.0 0.3.3
    Mandrake 8.0gcc-2.962.08.11.2 0.3.6
    RedHat 6.2egcs-1.1.22.2.28.11.6 0.5.4
    RedHat 7.1gcc-2.96?8.12.1 0.3.5
    RedHat 7.3gcc-2.962.2.28.11.6 0.5.5
    RedHat 7.3gcc-2.962.3.38.13.1 0.7.2
    RedHat 8.0gcc-3.22.2.18.12.6 0.5.2
    Debian Linuxgcc-2.95.22.1.18.12.0 0.3.7
    Debian Linuxgcc-3.2.22.2.28.12.7 0.5.4
    AIX-4.1.5gcc-2.95.22.1.18.11.5 0.3.3
    AIX-4.1.5gcc-2.95.22.1.18.12.1 0.3.4
    AIX-4.1.5gcc-2.95.22.1.38.12.3 0.4.2
    AIX-4.1.5gcc-2.95.22.2.38.13.1 0.7.1
    Slackware 7.1??8.12.1 0.3.8
    Slackware 9.0gcc-3.2.22.2.38.12.9 0.5.4
    OpenBSD?2.3.3?8.13.1? 0.7.2
    SuSE 7.3gcc-2.95.32.1.18.12.2 0.3.9
    FreeBSDgcc-2.95.32.2.18.12.3 0.4.0
    FreeBSDgcc-2.95.32.2.2? 0.5.5
    FreeBSD 4.4gcc-2.95.3?8.12.10 0.6.6

    Requirements

  • While the miltermodule will work with python 1.5, you probably want to use python 2.0 or better. The python code uses a number of python 2 features.
  • Python must be configured with thread support. This is because sendmail's libmilter requires thread support.
  • You must compile sendmail with libmilter enabled. In versions of sendmail prior to 8.12 libmilter is marked FFR (For Future Release) and is not installed by default. Sendmail 8.12 still does not enable libmilter by default. You must explicitly select the "MILTER" option when compiling.
  • Python milter has been tested against sendmail-8.11 and sendmail-8.12.
  • Python milter must be compiled for the specific version of sendmail it will run with. (Since the result is dynamically loaded, there could conceivably be multiple versions available and selected at startup - but that will have to wait.) This situation may only exist for sendmail versions prior to 8.12. The protocol seems designed for backward compatibility - and 8.12 is the first official milter release.
  • Mea Culpa! After reading the Python Style guide, I realize that my Python code is not up to snuff. Apparently mixed tabs and spaces are anathema to those using Windows editors, where tabs can be expanded using any arbitrary algorithm. Other than that, my intuition matched Guido's pretty well - although I like to indent by 2 rather than 4. I will arrange to have tabs expanded to spaces when exporting new versions. Until then, beware!
  • AIX 4.1.5 Requirements

    To create sendmail RPMs for AIX, you can download my AIX 4.1.5 spec files for sendmail-8.11.5 or sendmail-8.12.3. If you have not already set it up, I use a dummy RPM package to represent the stuff that comes with AIX. You might also want my python-2.1.1 spec file for AIX. It does not include Tk or curses modules, sorry. If y'all trust me, you can download rpms for AIX 4.x from my AIX RPM directory.

    Sendmail-8.12 renames libsmutil.a to libsm.a. Unfortunately, libsm.a is an important AIX system shared library. Therefore, I rename libsm.a back to libsmutil.a for AIX. This presents a problem for setup.py.

    RedHat 7.2 Requirements

    If you are running Redhat 7.2, the distributed version of sendmail now enables libmilter by default. RedHat 7.2 bundles the development libraries with the main sendmail package, so there is no sendmail-devel package. However, they forgot to include the headers! So you'll have to get the SRPM and modify it. I suggest moving the static libs to a devel package and adding the headers. If this is too much trouble, you can get the mfapi.h header for sendmail-8.6.11 from here and manually install it as /usr/include/libmilter/mfapi.h.

    If you do modify the SRPM, I suggest renaming libsmutil.a to libsm.a - just like sendmail-8.12 will. If you manually install mfapi.h or don't rename libsmutil.a, you'll need to force libs = ["milter", "smutil"] in setup.py.

    If you have installed python2, and want python-milter to use python2, add python=python2 to setup.cfg and build with python2 setup.py bdist_rpm.

    Redhat 6.2 Requirements

    If you are running Redhat 6.2, the distributed version of sendmail does not enable libmilter. You can download the Redhat 7.2 sendmail.spec modified to compile on RedHat 6.2: sendmail-rhmilter.spec. The SRPM for sendmail-8.11.6 is available from Redhat under Errata for RH6.2. But that doesn't include the latest security patches since RH6.2 is no longer supported.

    If y'all trust me, you can pick up source and binary sendmail RPMs for RH6.2 from my linux downloads directory. The lastest RPMs were built by taking a RH7.2 SRPMS and removing some RPM features from the spec file that RH6.2 doesn't support, then recompiling on RH6.2. You can check this by installing the RH7.2 SRPM, then diffing my sendmail.spec with theirs. Then run "rpm -bb sendmail-rhmilter.spec" when you are satisfied.

    If you have installed python2, and want python-milter to use python2, add python=python2 to setup.cfg and build with python2 setup.py bdist_rpm. You'll need to install the sendmail-devel package to compile milter.


     [ Valid HTML 3.2! ]  [ Powered By Red Hat Linux ]