Viewable With Any Browser Your vote? I Disagree I Agree

Sendmail Milters in Python

by Jim Niemira and Stuart D. Gathman
This web page is written by Stuart D. Gathman
and
sponsored by Business Management Systems, Inc.
Last updated Apr 05, 2004

See the FAQ | Download now | Subscribe to mailing list

A Python Sendmail introduced a new API beginning with version 8.10 - libmilter. The milter module for Python provides a python interface to libmilter that exploits all its features.

Sendmail 8.12 officially releases libmilter. Version 8.12 seems to be more robust, and includes new privilege separation features to enhance security. I recommend upgrading.

Bayesian Filtering

I have selected the dspam bayes filter project and packaged it for python. Release 0.6.0 offers a simple application of dspam I call "header triage", which rejects messages with spammy headers. Since sendmail has to read the entire message anyway once we start reading headers, it would probably be better to scan the whole message - except that we replace dangerous attachments elsewhere in the milter - which screws up the body statistics for messages with dangerous attachments.

Release 0.6.1 adds a full milter based dspam application.

To use header triage, you must have DSPAM installed, and select a dictionary that is well moderated by someone who gets lots of spam. That dictionary can be used to block spam that is obvious from the headers (e.g. X-Mailer and Subject) before it ties up any more resources. I have yet to see any false positives from this approach (check the milter log), but if there are, the sender will get a REJECT with the message "Your message looks spammy."

Enough Already!

Nearly a dozen people have emailed me begging for a feature to copy outgoing and/or incoming mail to a backup directory by user. Ok, it looks like this is a most requested feature for 0.5.6. In the meantime, here are some things to consider:

To Bcc a message, call self.add_recipient(rcpt) in envfrom after determining whether you want to copy (e.g. whether the sender is local). For example,

  def envfrom(...
    ...
    if len(t) == 2:
      self.rejectvirus = t[1] in reject_virus_from
      if t[0] in wiretap_users.get(t[1],()):
	self.add_recipient(wiretap_dest)
      if t[1] == 'mydomain.com':
        self.add_recipient('<copy-%s>' % t[0])
      ...

To make this a generic feature requires thinking about how the configuration would look. Feel free to make specific suggestions about config file entries. Be sure to handle both Bcc and file copies, and designating what mail should be copied. How should "outgoing" be defined? Implementing it is easy once the configuration is designed.

Overview

This package provides a robust toolkit for Python milters, and the beginnings of a general purpose mail filtering system written in Python.

At the lowest level, the 'milter' module provides a thin wrapper around the sendmail libmilter API. This API lets you register callbacks for a number of events in the process of sendmail receiving a message via SMTP. These events include the initial connection from a MTA, the envelope sender and recipients, the top level mail headers, and the message body. There are options to mangle all of these components of the message as it passes through the milter.

At the next level, the 'Milter' module (note the case difference) provides a Python friendly object oriented wrapper for the low level API. To use the Milter module, an application registers a 'factory' to create an object for each connection from a MTA to sendmail. These connection objects must provide methods corresponding to the libmilter callback events.

Each event method returns a code to tell sendmail whether to proceed with processing the message. This is a big advantage of milters over other mail filtering systems. Unwanted mail can be stopped in its tracks at the earliest possible point.

The Milter.Milter class provides default implementations for event methods that do nothing, and also provides wrappers for the libmilter methods to mutate the message.

Finally, the bms.py application is both a sample of how to use the Milter module, and the beginnings of a general purpose SPAM filtering, wiretapping, and Win32 virus protection milter.

Downloading

The latest stable release is 0.6.6. A stable release is one which has been installed (and working correctly) on production systems long enough to convince me that it is stable. As the package gains more features and complexity, stable will mean no bug reports from outside users either.

The latest version is 0.6.7. See the Change Log.

milter-0.6.7.tar.gz Explicit local socket bug, SRS forgery detection, thread resource starvation detection. SRS support requires pysrs.

milter-0.6.7-3.i386.rpm Binary RPM for Redhat 7.x, now requires sendmail-8.12 and python2.3.
milter-0.6.7-3.src.rpm Source RPM for Redhat 7.x. Release 0.6.7-3 patches:

Stable milter-0.6.6.tar.gz Plug another memory leak, SPF support, hello blacklist. SPF support requires pydns. NOTE - the spf.py module included is modified from the official 1.6 version at wayforward.net. I neglected to add the CVS log. The changes are expanded result codes and tolerating common method misspellings in SPF records. I have notified the author, but haven't heard back. At some point, the RPM will include the official pyspf tarball and apply patches.

milter-0.6.6-2.i386.rpm Binary RPM for Redhat 7.x, now requires sendmail-8.12 and python2.3. Release 2 fixes sysv init script bug for python2.3.
milter-0.6.6-2.src.rpm Source RPM for Redhat 7.x

milter-0.6.5.tar.gz Plug memory leak, progress reporting, trusted relay. Redhat RPM now requires sendmail-8.12.

milter-0.6.5-2.i386.rpm Binary RPM for Redhat 7.x
milter-0.6.5-2.src.rpm Source RPM for Redhat 7.x

milter-0.6.4.tar.gz Numerous Dspam fixes. Requires pydspam-1.1.5 and dspam-2.6.5.2 for Dspam features. The dspam-python RPM has been replaced by pydspam.

milter-0.6.4-1.i386.rpm Binary RPM for Redhat 7.x

milter-0.6.3.1.tar.gz New dspam SCREENER feature with pydspam-1.1.4. Don't save a defang copy of false positives. Fixed an oops from last fix, rejecting false positives. BUG: sendmail-8.11 doesn't invoke milter when sending mail via sendmail from command line (8.12 works). Therefore, the supplied falsepositive script for milter based dspam doesn't work with stock RedHat 7.x. I am writing a HOWTO for configuring milter based dspam that will address this (and a fix in the next version).

milter-0.6.3-1.i386.rpm Binary RPM for Redhat 7.x

milter-0.6.2.tar.gz work around email.Message.get_filename bug, dspam_exempt list, REJECT messages with missing MIME boundaries (which are almost always spam), DISCARD messages which any dspam user flags as spam, start.sh was calling python instead of python2 on Linux.

milter-0.6.2-1.src.rpm Source RPM for Redhat 7.x (and likely higher versions)

milter-0.6.1.tar.gz dspam milter application, python-2.2.3 support.

You must have dspam and dspam-python loaded for the dspam feature to work. Brief instructions for configuring are in the default config file. This is working at a customer, but I'm sure a few more iterations will be required to make setup as smooth as possible.

NOTE: Outlook destroys dspam tags when forwarding mail (while converting HTML to text). Perhaps some config option will turn this abominable "feature" off. Working around this by making dspam tags visble on HTML mail is ugly. My suggestion is to not use Outlook, for this and many other reasons - especially security. Any other suggestions for those married to Microsoft are welcome. The DSPAM LDA works around this by making the tags visible in HTML attachments. This is ugly, and occasionally corrupts attachments.

We have to supply workarounds for bugs in the email module (reported to sourceforge). The workarounds reference some internal variables which change with python versions.

milter-0.6.1-1.i386.rpm Binary RPM for Redhat 7.x

milter-0.6.1-1.src.rpm Source RPM for Redhat 7.x (and likely higher versions)

milter-0.6.0.tar.gz simple dspam pre-filtering, use email module, requires python >= 2.2.2.

milter-0.6.0-1.i386.rpm Binary RPM for Redhat 7.x

milter-0.6.0-1.src.rpm Source RPM for Redhat 7.x (and likely higher versions)

milter-0.5.5.tar.gz IPV6 support, passing None to set_XXX_callback, set_reply, chg_header, detect internal connections. Note, this release did not work on AIX4.1.5, probably due to IPV6 support breaking something. The milter.so module from 0.5.4 can be installed to use this release with AIX.

milter-0.5.4.tar.gz wiretap, smart alias features, quarantine support.

The name of the production "sample" milter "bms.py" now stands for "Basic Milter System" until someone suggests a better name. The test coverage is rather sparse at present. Please email with proposals for what to name the milter application.

NOTES

milter-0.5.2.tar.gz Fix and unittest another HTML parsing bug.
milter-0.5.1.tar.gz Handle encoded rfc822 attachments.
milter-0.5.0.tar.gz Use a config file so users don't have to keep syncing with bms.py.
milter-0.4.5.tar.gz Work with sgmlop. Reduce local hacks to config variables.

Python milter is under GPL. The authors can probably be convinced to change this to LGPL.

What is a milter?

Milters can run on the same machine as sendmail, or another machine. The milter can even run with a different operating system or processor than sendmail. Sendmail talks to the milter via a local or internet socket. Sendmail keeps the milter informed of events as it processes a mail connection. At any point, the milter can cut the conversation short by telling sendmail to ACCEPT, REJECT, or DISCARD the message. After receiving a complete message from sendmail, the milter can again REJECT or DISCARD it, but it can also ACCEPT it with changes to the headers or body.

What can you do with a milter?

  • A milter can DISCARD or REJECT spam based based on algorithms scripted in python rather than sendmail's cryptic "cf" language.
  • A milter can alter or remove attachments from mail that are poisonous to Windows.
  • A milter can scan for viruses and clean them when detected.
  • A milter scans outgoing as well as incoming mail.
  • A milter can add and delete recipients to forward or secretly copy mail.
  • For more ideas, check the Milter Web Page.
  • Documentation for the C API is provided with sendmail. Miltermodule provides a thin python wrapper for the C API. Milter.py provides a simple OO wrapper on top of that.

    The Python milter package includes a sample milter that replaces dangerous attachments with a warning message, discards mail addressed to MAILER-DAEMON, and demonstrates several SPAM abatement strategies. The MimeMessage class to do this used to be based on the mimetools and multifile standard python packages. As of milter version 0.6.0, it is based on the email standard python packages, which were derived from the mimelib project. The MimeMessage class patches several bugs in the email package, and provides some backward compatibility.

    The "defang" function of the sample milter was inspired by MIMEDefang, a Perl milter with flexible attachment processing options. The latest version of MIMEDefang uses an apache style process pool to avoid reloading the Perl interpreter for each message. This makes it fast enough for production and does not use Perl threading.

    mailchecker is a Python project to provide flexible attachment processing for mail. I will be looking at plugging mailchecker into a milter.

    TMDA is a Python project to require confirmation the first time someone tries to send to your mailbox. This would be a nice feature to have in a milter.

    There is also a Milter community website where milter software and gory details of the API are discussed.

    Is a milter written in python efficient?

    The python milter process is multi-threaded and startup cost is incurred only once. This is much more efficient than some implementations that start a new interpreter for each connection. Testing in a production environment did not use a significant percentage of the CPU. Furthermore, python is easily extended in C for any step requiring expensive CPU processing.

    For example, the HTML parsing feature to remove scripts from HTML attachments is rather CPU intensive in pure python. Using the C replacement for sgmllib greatly speeds things up.

    Goals

  • Implement RRS - a backdoor for non-SRS forwarders. User lists non-SRS forwarder accounts (perhaps in ~/.forwarders), and a util provides a special local alias for the user to give to the forwarder. Alias only works for mail from that forwarder. Milter gets forwarder domain from alias and uses it to SPF check forwarder. Requires milter to have read access to ~/.forwarders or else a way for user to submit entries to milter database.
  • The bms.py milter has too many features. Create a framework where numerous small feature modules can be plugged together in the configuration.
  • Create a pure python substitute for miltermodule and libmilter that implements the libmilter protocol in python.
  • Find or write a faster implementation of sgmllib. The sgmlop package is not very compatible with Python-2.1 sgmllib, but it is a start, and is supported in milter-0.4.5 or later.
  • Implement all or most of the features of MIMEDefang.
  • Follow the official Python coding standards more closely.
  • Make unit test code more like other python modules.
  • Confirmed Installations

    Please email me if you successfully install milter on a system not mentioned below.

    Operating System Compiler Python Sendmail milter
    Mandrake 8.0gcc-3.0.12.1.18.12.0 0.3.3
    Mandrake 8.0gcc-2.962.08.11.2 0.3.6
    RedHat 6.2egcs-1.1.22.2.28.11.6 0.5.4
    RedHat 7.1gcc-2.96?8.12.1 0.3.5
    RedHat 7.2gcc-2.962.1.18.11.6 0.4.1
    RedHat 7.2gcc-2.962.2.18.11.6 0.4.5
    RedHat 7.2gcc-2.962.2.28.11.6 0.5.5
    RedHat 7.2gcc-2.962.3.38.12.10 0.6.6
    RedHat 7.3gcc-2.962.2.28.11.6 0.5.5
    RedHat 7.3gcc-2.962.3.38.12.10 0.6.6
    RedHat 8.0gcc-3.22.2.18.12.6 0.5.2
    Debian Linuxgcc-2.95.22.1.18.12.0 0.3.7
    Debian Linuxgcc-3.2.22.2.28.12.7 0.5.4
    AIX-4.1.5gcc-2.95.22.1.18.11.5 0.3.3
    AIX-4.1.5gcc-2.95.22.1.18.12.1 0.3.4
    AIX-4.1.5gcc-2.95.22.1.38.12.3 0.4.2
    AIX-4.1.5gcc-2.95.22.2.28.12.6 0.5.4
    Slackware 7.1??8.12.1 0.3.8
    Slackware 9.0gcc-3.2.22.2.38.12.9 0.5.4
    OpenBSD?2.1.18.11.6 0.3.9
    SuSE 7.3gcc-2.95.32.1.18.12.2 0.3.9
    FreeBSDgcc-2.95.32.2.18.12.3 0.4.0
    FreeBSDgcc-2.95.32.2.2? 0.5.5
    FreeBSD 4.4gcc-2.95.3?8.12.10 0.6.6

    Requirements

  • While the miltermodule will work with python 1.5, you probably want to use python 2.0 or better. The python code uses a number of python 2 features.
  • Python must be configured with thread support. This is because sendmail's libmilter requires thread support.
  • You must compile sendmail with libmilter enabled. In versions of sendmail prior to 8.12 libmilter is marked FFR (For Future Release) and is not installed by default. Sendmail 8.12 still does not enable libmilter by default. You must explicitly select the "MILTER" option when compiling.
  • Python milter has been tested against sendmail-8.11 and sendmail-8.12.
  • Python milter must be compiled for the specific version of sendmail it will run with. (Since the result is dynamically loaded, there could conceivably be multiple versions available and selected at startup - but that will have to wait.) This situation may only exist for sendmail versions prior to 8.12. The protocol seems designed for backward compatibility - and 8.12 is the first official milter release.
  • Mea Culpa! After reading the Python Style guide, I realize that my Python code is not up to snuff. Apparently mixed tabs and spaces are anathema to those using Windows editors, where tabs can be expanded using any arbitrary algorithm. Other than that, my intuition matched Guido's pretty well - although I like to indent by 2 rather than 4. I will arrange to have tabs expanded to spaces when exporting new versions. Until then, beware!
  • AIX 4.1.5 Requirements

    To create sendmail RPMs for AIX, you can download my AIX 4.1.5 spec files for sendmail-8.11.5 or sendmail-8.12.3. If you have not already set it up, I use a dummy RPM package to represent the stuff that comes with AIX. You might also want my python-2.1.1 spec file for AIX. It does not include Tk or curses modules, sorry. If y'all trust me, you can download rpms for AIX 4.x from my AIX RPM directory.

    Sendmail-8.12 renames libsmutil.a to libsm.a. Unfortunately, libsm.a is an important AIX system shared library. Therefore, I rename libsm.a back to libsmutil.a for AIX. This presents a problem for setup.py.

    RedHat 7.2 Requirements

    If you are running Redhat 7.2, the distributed version of sendmail now enables libmilter by default. RedHat 7.2 bundles the development libraries with the main sendmail package, so there is no sendmail-devel package. However, they forgot to include the headers! So you'll have to get the SRPM and modify it. I suggest moving the static libs to a devel package and adding the headers. If this is too much trouble, you can get the mfapi.h header for sendmail-8.6.11 from here and manually install it as /usr/include/libmilter/mfapi.h.

    If you do modify the SRPM, I suggest renaming libsmutil.a to libsm.a - just like sendmail-8.12 will. If you manually install mfapi.h or don't rename libsmutil.a, you'll need to force libs = ["milter", "smutil"] in setup.py.

    If you have installed python2, and want python-milter to use python2, add python=python2 to setup.cfg and build with python2 setup.py bdist_rpm.

    Redhat 6.2 Requirements

    If you are running Redhat 6.2, the distributed version of sendmail does not enable libmilter. You can download the Redhat 7.2 sendmail.spec modified to compile on RedHat 6.2: sendmail-rhmilter.spec. The SRPM for sendmail-8.11.6 is available from Redhat under Errata for RH6.2. But that doesn't include the latest security patches since RH6.2 is no longer supported.

    If y'all trust me, you can pick up source and binary sendmail RPMs for RH6.2 from my linux downloads directory. The lastest RPMs were built by taking a RH7.2 SRPMS and removing some RPM features from the spec file that RH6.2 doesn't support, then recompiling on RH6.2. You can check this by installing the RH7.2 SRPM, then diffing my sendmail.spec with theirs. Then run "rpm -bb sendmail-rhmilter.spec" when you are satisfied.

    If you have installed python2, and want python-milter to use python2, add python=python2 to setup.cfg and build with python2 setup.py bdist_rpm. You'll need to install the sendmail-devel package to compile milter.


     [ Valid HTML 3.2! ]  [ Powered By Red Hat Linux ]