diff --git a/doc/changes.ht b/doc/changes.ht
index a0eaa4afaafbc43dac8ad7c99acc8414e22bb8fe..719d09b922812c1f2de7da55d2ea12dde05e965f 100644
--- a/doc/changes.ht
+++ b/doc/changes.ht
@@ -2,6 +2,47 @@ Title: Recent Changes
<h2> Recent Changes </h2>
+<h3> 0.8.10 </h3>
+
+SRS rejections now log the recipient.
+I have finally implemented plain CBV (no DSN). The CBV policy
+will do a plain CBV from now on, and the DSN policy is required
+if you want to send a DSN.
+I started checking the MAIL FROM fullname (human readable part
+of an email) for porn keywords. There is now a banned IP database.
+IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned
+for 7 days.
+
+<h3> 0.8.9 </h3>
+
+I use the <code>%ifarch</code> hack to build milter and milter-spf
+packages as noarch, while pymilter is built as native.
+
+I removed the spf dependency from dsn.py, so pymilter can be used without
+installing pyspf, and added a Milter.dns module to let python milters do
+general DNS lookups without loading pyspf.
+
+<h3> 0.8.8 </h3>
+
+Programs do not belong in the /var/log directory. I moved the
+milter apps to /usr/lib/pymilter. Since having the programs and
+data in the same directory is convenient for debugging, it will
+still use an executable present in the datadir.
+
+Several general utility classes and functions are now in the Milter package
+for possible use by other python milters. In addition to the trivial example
+milter, a simple SPF only milter is included as a realistic example.
+
+The spec file now build 3 RPMs:
+
+<ul>
+<li> pymilter is the milter module and Milter package for use by all python
+ milters.
+<li> milter is the all-singing, all-dancing python milter application, with
+ supporting <code>/etc/init.d</code>, logrotate and other scripts.
+<li> milter-spf is the simple SPF only milter application.
+</ul>
+
<h3> 0.8.7 </h3>
The spf module has been moved to the
diff --git a/doc/credits.ht b/doc/credits.ht
index 849ae3dcc3bc579dd602378710a6ce67d4e7e7b4..9973a4df9956ac2fe338643df55e15b3a5789d1e 100644
--- a/doc/credits.ht
+++ b/doc/credits.ht
@@ -5,7 +5,7 @@ Title: Credits
<a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a>
wrote the original C module and some quick
and dirty python to use it.
-<a href="mailto:Stuart Gathman <stuart@bmsi.com>">Stuart D. Gathman</a>
+<a href="http://gathman.org/vitae">Stuart D. Gathman</a>
took that kludge and added threading and context objects to it, wrote a proper
OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged
it with distutils, and generally transformed it from a quick hack to a
diff --git a/doc/policy.ht b/doc/policy.ht
index c559cb119e61816a72a1dd0963a1b21400fe044d..750cf087dc8e624f044791ab9e0b5e26bfdbd4be 100644
--- a/doc/policy.ht
+++ b/doc/policy.ht
@@ -4,8 +4,7 @@ Title: Python Milter Mail Policy
These are the policies implemented by the <code>bms.py</code> milter
application. The milter and Milter modules do not implement any policies
-by themselves. Eventually, I'll get the bms.py milter moved to its
-own package.
+by themselves.
<h3> Classify connection </h3>
@@ -77,161 +76,119 @@ altered accordingly.
<h2> SPF check </h2>
-Finally, the MAIL FROM, connect IP, and HELO name are checked against
-any SPF records published via DNS for the alleged sender (MAIL FROM).
-If there is no SPF record, we check for a local substitute under the
-domain defined in the <code>[spf]delegate</code> configuration.
-Further checks depend on the result.
-
-<table border=1>
-<tr><th>NONE</th><td>
-If there is no SPF record (official or delegated), then we
-initiate a "three strikes and your out" regime, which looks for
-<b>some</b> form of validated identification.
-<ol>
-<li>We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr". If this
- passes, good.
-<li> We try to validate the HELO name. First check for an SPF record.
- Otherwise, check whether the connect IP matches any A record for
- the HELO name, or any A record for any MX name for the HELO name,
- or is at least in the same /24 subnet as any of the above.
- (In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".)
- If so, good. We consider the HELO validated. If the HELO SPF
- check fails, we reject the email.
-</ol>
+The MAIL FROM, connect IP, and HELO name are checked against
+any SPF records published via DNS for the alleged sender (MAIL FROM)
+to determine the official SPF policy result.
+The offical SPF result is then logged in the Received-SPF header field,
+but certain results are subjected to further processing to create
+an effective result for policy purposes.
+
+If the official result is 'none', we try to turn it into an effective result of
+'pass' or 'fail'. First, we check for a local substitute SPF record
+under the domain defined in the <code>[spf]delegate</code> configuration.
+It is often useful to add local SPF records for correspondents that are
+too clueless to add their own. If there is no local substitute, we use a "best
+guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24
+mx/24" for HELO. In addition, a HELO that is a subdomain of MAIL FROM and
+resolves to the connect IP results in an effective result of 'pass'.
+
+If there is no local SPF record, and the effective result is still not
+'pass', we check for either a valid HELO name or a valid PTR record for
+the connect IP. A valid HELO or PTR cannot look like a dynamic name
+as determined by the heuristic in <code>Milter.dynip</code>.
+
+If HELO has an SPF record, and the result is anything but pass, we reject
+the connection:
<pre>
2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
2005Jul30 19:45:18 [93991] hello from adelphia.net
2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> ()
2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
</pre>
-<ol>
-<li> If there is a validated PTR name, and it doesn't look
- like a dynamic name, good. We consider the connection validated.
-</ol>
-If any of the above can be validated, we continue on.
-If none of the above can be validated, and the <code>[SPF]reject_noptr</code>
-option is true, we reject the message immediately with the explanation
-that we need some form of valid identification before we accept an email.
-If <code>[SPF]reject_noptr</code> is false, we flag the message as
-needing Call Back Validation.
-The Call Back Valildation sends a DSN to the purported sender informing
-them of the lack of identification. If the message is legitimate, the
-sender needs to know that their email setup is broken and should be corrected.
-If the message is forged, the sender is informed of the forgery,
-and their need to publish an SPF record or at least use a valid HELO name.
-If the purported sender does not accept the DSN,
-then the message is rejected. The CBV status is cached to avoid
-annoying the purported sender with too many DSNs. Currently, the DSN
-is repeated to the same sender once per month.
-<p>
-In this example, although 3com.com has no SPF record, we assume that
-any legitimate mail from them will at least have a valid HELO or PTR.
-<pre>
-2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN
-2005Jul30 23:52:03 [96777] hello from 3mail.3com.com
-2005Jul30 23:52:04 [96777] mail from <etec_nic_family@3mail.3com.com> ()
-2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF
-</pre>
-</td></tr>
+Note that HELO does not have any forwarding issues like MAIL FROM, and so
+any result other than 'pass' or 'none' should be treated like 'fail'.
-<tr><th>PASS</th><td>
-A pass result normally lets the email continue on, but the domain is
-tracked for reputation (and may be blocked), and may skip content scanning if
-it matches a whitelist.
-<pre>
-2005Jul24 17:44:26 [2104] mail from <gnucash-devel-bounces@gnucash.org> ('SIZE=4410',)
-2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org
- designates 204.107.200.65 as permitted sender)
- client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org;
-</pre>
-</td></tr>
+Only if nothing about the SMTP envelope can be validated does the effective
+result remain 'none. I call this the "3 strikes" rule.
-<tr><th>NEUTRAL</th><td>
-A neutral result normally lets the email continue on, but the domain is not
-tracked for reputation or matched against any whitelists.
-Highly forged domains listed in <code>[SPF]reject_neutral</code> are
-rejected.
-<pre>
-2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL
-2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl
-2005Jul24 17:41:38 [2070] mail from <nwarjejkw@yahoo.com> ()
-2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com
-</pre>
-</td></tr>
+If the official result is 'permerror' (a syntax error in the sender's
+policy), we use the 'lax' option in pyspf to try various heuristics to guess
+what they really meant. For instance, the invalid mechanism "ip:1.2.3.4" is
+treated as "ip4:1.2.3.4". The result of lax processing is then used
+as the effective result for policy purposes.
-<tr><th>SOFTFAIL</th><td>
-A softfail result normally lets the email continue on, but the domain is not
-tracked for reputation or matched against any whitelists. Furthermore,
-the message is flagged as needing Call Back Validation,
-and the highly forged domains listed in <code>[SPF]reject_neutral</code> are
-rejected as well.
-<p>
-At present, we also require a valid HELO or PTR to avoid rejecting
-a softfail. But this should probably change to only require a
-successful CBV.
-<p>
-The Call Back Valildation sends a DSN to the purported sender informing
-them of the softfail. If the message is legitimate, the sender needs
-to know about the softfail so that their email setup can be corrected.
-If the message is forged, the sender is informed of the forgery, confirming
-that SPF is protecting their reputation and encouraging a rapid transition
-to a strict policy. If the purported sender does not accept the DSN,
-then the message is rejected. The CBV status is cached to avoid
-annoying the purported sender with too many DSNs. Currently, the DSN
-is repeated to the same sender once per month.
-<pre>
-2005Jul24 15:41:33 [801] mail from <Aitp@horafeliz.com> ()
-2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com
- does not designate 221.184.83.185 as permitted sender)
- client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com;
- helo=p8185-ipad30funabasi.chiba.ocn.ne.jp;
-2005Jul24 15:41:33 [801] rcpt to <david@example.com> ()
-2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount.
-2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605
-2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com
-2005Jul24 15:41:38 [801] REJECT: CBV: 550 <Aitp@horafeliz.com>: User unknown
-</pre>
-</td></tr>
+With an effective SPF result in hand, we consult the sendmail access
+database to find our receiver policy for the sender.
-<tr><th>FAIL</th><td>
-The message is rejected with a reference the SPF why page.
-<pre>
-2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN
-2005Jul30 19:53:27 [94070] hello from winzip.com
-2005Jul30 19:53:27 [94070] mail from <dan@winzip.com> ()
-2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail:
- see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16
-</pre>
+<table border=1>
+<tr><th>REJECT</th><td>
+Reject the sender with a 550 5.7.1 SMTP code. The SMTP rejection
+includes a detailed description of the problem.
+</td></tr>
+<tr><th>CBV</th><td>
+Do a Call Back Validation by connecting to an MX of the sender
+and checking that using the sender as the RCPT TO is not rejected.
+We quit the CBV connection before actualling sending a message.
+If the CBV is rejected, our SMTP connection is rejected with the
+same error code and message. CBV results are cached.
+</td></tr>
+<tr><th>DSN</th><td>
+Do a Call Back Validation by connecting to an MX of the sender
+and checking that using the sender as the RCPT TO is not rejected.
+Unlike a CBV, we continue on to data and send a detailed message
+explaining the problem. This can be useful for reporting PermError
+or SoftFail to the sender. Keep in mind that for any result other
+than 'pass', the sender could be forged, and your DSN could annoy the
+wrong person. However, a SoftFail result is requesting such feedback
+for debugging and a PermError result needs to be fixed by the sender ASAP
+whether forged or not. DSN results are cached so that senders are
+annoyed only weekly.
</td></tr>
+<tr><th>OK</th><td>
+Accept the sender. The message may still be rejected via reputation
+or content filtering.
+</td></tr>
+</table>
-<tr><th>PERMERROR</th><td>
-Permanent errors were called "unknown", and are still show that way
-in the log. The message is rejected. Previously, we enabled "lax" parsing
-of the SPF record, but rejecting is better because it informs the
-sender about their problem. The next milter version will
-look for a local substitute SPF record (as for a missing SPF record)
-before rejecting. This will inform the sender of their problem, but
-also let the receiver install a temporary workaround.
+<h3> SPF policy syntax </h3>
+
+First, the full sender is checked:
<pre>
-2005Jul24 18:05:37 [2312] mail from <b-mihdbcgaacaa-becibijh-000-@msg.euxiphipops.com> ()
-2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error:
- include mechanism missing domain: include
+SPF-Fail:abeb@adelphia.net DSN
</pre>
-The SPF record for msg.euxiphipops.com looked like this at the time of the
-above error:
+This says to accept mail from that adelphia.net user despite the
+SPF fail, but only after annoying them with a DSN about their ISP's broken
+policy.
+
+If there is no match on the full sender, the domain is checked:
<pre>
-msg.euxiphipops.com TXT "v=spf1 mx ptr a include"
+SPF-Neutral:aol.com REJECT
</pre>
-</td></tr>
+This says to reject mail from AOL with an SPF result of neutral.
+This means AOL users can't use their AOL address with another mail service
+to send us mail. This is good because the other mail service is
+likely a badly configured greeting card site or a virus.
-<tr><th>TEMPERROR</th><td>
-Temporary errors result in a 451 "Try again later" response. The sender
-should retry the message at a later time.
+Finally, a default policy for the result is checked. While there are program
+defaults, you should have defaults in the access database for SPF results:
<pre>
-2005Jul24 07:33:13 [29846] mail from <quickenloans@rate.quicken.com> ('SIZE=73775', 'BODY=8BITMIME')
-2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout
+SPF-Neutral: CBV
+SPF-Softfail: DSN
+SPF-PermError: DSN
+SPF-TempError: REJECT
+SPF-None: REJECT
+SPF-Fail: REJECT
+SPF-Pass: OK
</pre>
-</td></tr>
-</table>
+<h2> Reputation </h2>
+
+If the sender has not been rejected by this point, and if a GOSSiP server is
+configured, we consult GOSSiP for the reputation score of the sender and
+SPF result. The score is a number from -100 to 100 with a confidence
+percentage from 0 to 100. A really bad reputation (less than -50 with
+confidence greater than 3) is rejected. Note that the reputation is tracked
+independently for each SPF result and sender combination. So aol.com:neutral
+might have a really bad reputation, while aol.com:pass would be ok.
+Furthermore, when a sender finally publishes an SPF policy and starts
+getting SPF pass, their reputation is effectively reset.