diff --git a/doc/changes.ht b/doc/changes.ht index a0eaa4afaafbc43dac8ad7c99acc8414e22bb8fe..719d09b922812c1f2de7da55d2ea12dde05e965f 100644 --- a/doc/changes.ht +++ b/doc/changes.ht @@ -2,6 +2,47 @@ Title: Recent Changes <h2> Recent Changes </h2> +<h3> 0.8.10 </h3> + +SRS rejections now log the recipient. +I have finally implemented plain CBV (no DSN). The CBV policy +will do a plain CBV from now on, and the DSN policy is required +if you want to send a DSN. +I started checking the MAIL FROM fullname (human readable part +of an email) for porn keywords. There is now a banned IP database. +IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned +for 7 days. + +<h3> 0.8.9 </h3> + +I use the <code>%ifarch</code> hack to build milter and milter-spf +packages as noarch, while pymilter is built as native. + +I removed the spf dependency from dsn.py, so pymilter can be used without +installing pyspf, and added a Milter.dns module to let python milters do +general DNS lookups without loading pyspf. + +<h3> 0.8.8 </h3> + +Programs do not belong in the /var/log directory. I moved the +milter apps to /usr/lib/pymilter. Since having the programs and +data in the same directory is convenient for debugging, it will +still use an executable present in the datadir. + +Several general utility classes and functions are now in the Milter package +for possible use by other python milters. In addition to the trivial example +milter, a simple SPF only milter is included as a realistic example. + +The spec file now build 3 RPMs: + +<ul> +<li> pymilter is the milter module and Milter package for use by all python + milters. +<li> milter is the all-singing, all-dancing python milter application, with + supporting <code>/etc/init.d</code>, logrotate and other scripts. +<li> milter-spf is the simple SPF only milter application. +</ul> + <h3> 0.8.7 </h3> The spf module has been moved to the diff --git a/doc/credits.ht b/doc/credits.ht index 849ae3dcc3bc579dd602378710a6ce67d4e7e7b4..9973a4df9956ac2fe338643df55e15b3a5789d1e 100644 --- a/doc/credits.ht +++ b/doc/credits.ht @@ -5,7 +5,7 @@ Title: Credits <a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a> wrote the original C module and some quick and dirty python to use it. -<a href="mailto:Stuart Gathman <stuart@bmsi.com>">Stuart D. Gathman</a> +<a href="http://gathman.org/vitae">Stuart D. Gathman</a> took that kludge and added threading and context objects to it, wrote a proper OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged it with distutils, and generally transformed it from a quick hack to a diff --git a/doc/policy.ht b/doc/policy.ht index c559cb119e61816a72a1dd0963a1b21400fe044d..750cf087dc8e624f044791ab9e0b5e26bfdbd4be 100644 --- a/doc/policy.ht +++ b/doc/policy.ht @@ -4,8 +4,7 @@ Title: Python Milter Mail Policy These are the policies implemented by the <code>bms.py</code> milter application. The milter and Milter modules do not implement any policies -by themselves. Eventually, I'll get the bms.py milter moved to its -own package. +by themselves. <h3> Classify connection </h3> @@ -77,161 +76,119 @@ altered accordingly. <h2> SPF check </h2> -Finally, the MAIL FROM, connect IP, and HELO name are checked against -any SPF records published via DNS for the alleged sender (MAIL FROM). -If there is no SPF record, we check for a local substitute under the -domain defined in the <code>[spf]delegate</code> configuration. -Further checks depend on the result. - -<table border=1> -<tr><th>NONE</th><td> -If there is no SPF record (official or delegated), then we -initiate a "three strikes and your out" regime, which looks for -<b>some</b> form of validated identification. -<ol> -<li>We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr". If this - passes, good. -<li> We try to validate the HELO name. First check for an SPF record. - Otherwise, check whether the connect IP matches any A record for - the HELO name, or any A record for any MX name for the HELO name, - or is at least in the same /24 subnet as any of the above. - (In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".) - If so, good. We consider the HELO validated. If the HELO SPF - check fails, we reject the email. -</ol> +The MAIL FROM, connect IP, and HELO name are checked against +any SPF records published via DNS for the alleged sender (MAIL FROM) +to determine the official SPF policy result. +The offical SPF result is then logged in the Received-SPF header field, +but certain results are subjected to further processing to create +an effective result for policy purposes. + +If the official result is 'none', we try to turn it into an effective result of +'pass' or 'fail'. First, we check for a local substitute SPF record +under the domain defined in the <code>[spf]delegate</code> configuration. +It is often useful to add local SPF records for correspondents that are +too clueless to add their own. If there is no local substitute, we use a "best +guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24 +mx/24" for HELO. In addition, a HELO that is a subdomain of MAIL FROM and +resolves to the connect IP results in an effective result of 'pass'. + +If there is no local SPF record, and the effective result is still not +'pass', we check for either a valid HELO name or a valid PTR record for +the connect IP. A valid HELO or PTR cannot look like a dynamic name +as determined by the heuristic in <code>Milter.dynip</code>. + +If HELO has an SPF record, and the result is anything but pass, we reject +the connection: <pre> 2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN 2005Jul30 19:45:18 [93991] hello from adelphia.net 2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> () 2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied </pre> -<ol> -<li> If there is a validated PTR name, and it doesn't look - like a dynamic name, good. We consider the connection validated. -</ol> -If any of the above can be validated, we continue on. -If none of the above can be validated, and the <code>[SPF]reject_noptr</code> -option is true, we reject the message immediately with the explanation -that we need some form of valid identification before we accept an email. -If <code>[SPF]reject_noptr</code> is false, we flag the message as -needing Call Back Validation. -The Call Back Valildation sends a DSN to the purported sender informing -them of the lack of identification. If the message is legitimate, the -sender needs to know that their email setup is broken and should be corrected. -If the message is forged, the sender is informed of the forgery, -and their need to publish an SPF record or at least use a valid HELO name. -If the purported sender does not accept the DSN, -then the message is rejected. The CBV status is cached to avoid -annoying the purported sender with too many DSNs. Currently, the DSN -is repeated to the same sender once per month. -<p> -In this example, although 3com.com has no SPF record, we assume that -any legitimate mail from them will at least have a valid HELO or PTR. -<pre> -2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN -2005Jul30 23:52:03 [96777] hello from 3mail.3com.com -2005Jul30 23:52:04 [96777] mail from <etec_nic_family@3mail.3com.com> () -2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF -</pre> -</td></tr> +Note that HELO does not have any forwarding issues like MAIL FROM, and so +any result other than 'pass' or 'none' should be treated like 'fail'. -<tr><th>PASS</th><td> -A pass result normally lets the email continue on, but the domain is -tracked for reputation (and may be blocked), and may skip content scanning if -it matches a whitelist. -<pre> -2005Jul24 17:44:26 [2104] mail from <gnucash-devel-bounces@gnucash.org> ('SIZE=4410',) -2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org - designates 204.107.200.65 as permitted sender) - client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org; -</pre> -</td></tr> +Only if nothing about the SMTP envelope can be validated does the effective +result remain 'none. I call this the "3 strikes" rule. -<tr><th>NEUTRAL</th><td> -A neutral result normally lets the email continue on, but the domain is not -tracked for reputation or matched against any whitelists. -Highly forged domains listed in <code>[SPF]reject_neutral</code> are -rejected. -<pre> -2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL -2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl -2005Jul24 17:41:38 [2070] mail from <nwarjejkw@yahoo.com> () -2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com -</pre> -</td></tr> +If the official result is 'permerror' (a syntax error in the sender's +policy), we use the 'lax' option in pyspf to try various heuristics to guess +what they really meant. For instance, the invalid mechanism "ip:1.2.3.4" is +treated as "ip4:1.2.3.4". The result of lax processing is then used +as the effective result for policy purposes. -<tr><th>SOFTFAIL</th><td> -A softfail result normally lets the email continue on, but the domain is not -tracked for reputation or matched against any whitelists. Furthermore, -the message is flagged as needing Call Back Validation, -and the highly forged domains listed in <code>[SPF]reject_neutral</code> are -rejected as well. -<p> -At present, we also require a valid HELO or PTR to avoid rejecting -a softfail. But this should probably change to only require a -successful CBV. -<p> -The Call Back Valildation sends a DSN to the purported sender informing -them of the softfail. If the message is legitimate, the sender needs -to know about the softfail so that their email setup can be corrected. -If the message is forged, the sender is informed of the forgery, confirming -that SPF is protecting their reputation and encouraging a rapid transition -to a strict policy. If the purported sender does not accept the DSN, -then the message is rejected. The CBV status is cached to avoid -annoying the purported sender with too many DSNs. Currently, the DSN -is repeated to the same sender once per month. -<pre> -2005Jul24 15:41:33 [801] mail from <Aitp@horafeliz.com> () -2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com - does not designate 221.184.83.185 as permitted sender) - client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com; - helo=p8185-ipad30funabasi.chiba.ocn.ne.jp; -2005Jul24 15:41:33 [801] rcpt to <david@example.com> () -2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount. -2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605 -2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com -2005Jul24 15:41:38 [801] REJECT: CBV: 550 <Aitp@horafeliz.com>: User unknown -</pre> -</td></tr> +With an effective SPF result in hand, we consult the sendmail access +database to find our receiver policy for the sender. -<tr><th>FAIL</th><td> -The message is rejected with a reference the SPF why page. -<pre> -2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN -2005Jul30 19:53:27 [94070] hello from winzip.com -2005Jul30 19:53:27 [94070] mail from <dan@winzip.com> () -2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail: - see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16 -</pre> +<table border=1> +<tr><th>REJECT</th><td> +Reject the sender with a 550 5.7.1 SMTP code. The SMTP rejection +includes a detailed description of the problem. +</td></tr> +<tr><th>CBV</th><td> +Do a Call Back Validation by connecting to an MX of the sender +and checking that using the sender as the RCPT TO is not rejected. +We quit the CBV connection before actualling sending a message. +If the CBV is rejected, our SMTP connection is rejected with the +same error code and message. CBV results are cached. +</td></tr> +<tr><th>DSN</th><td> +Do a Call Back Validation by connecting to an MX of the sender +and checking that using the sender as the RCPT TO is not rejected. +Unlike a CBV, we continue on to data and send a detailed message +explaining the problem. This can be useful for reporting PermError +or SoftFail to the sender. Keep in mind that for any result other +than 'pass', the sender could be forged, and your DSN could annoy the +wrong person. However, a SoftFail result is requesting such feedback +for debugging and a PermError result needs to be fixed by the sender ASAP +whether forged or not. DSN results are cached so that senders are +annoyed only weekly. </td></tr> +<tr><th>OK</th><td> +Accept the sender. The message may still be rejected via reputation +or content filtering. +</td></tr> +</table> -<tr><th>PERMERROR</th><td> -Permanent errors were called "unknown", and are still show that way -in the log. The message is rejected. Previously, we enabled "lax" parsing -of the SPF record, but rejecting is better because it informs the -sender about their problem. The next milter version will -look for a local substitute SPF record (as for a missing SPF record) -before rejecting. This will inform the sender of their problem, but -also let the receiver install a temporary workaround. +<h3> SPF policy syntax </h3> + +First, the full sender is checked: <pre> -2005Jul24 18:05:37 [2312] mail from <b-mihdbcgaacaa-becibijh-000-@msg.euxiphipops.com> () -2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error: - include mechanism missing domain: include +SPF-Fail:abeb@adelphia.net DSN </pre> -The SPF record for msg.euxiphipops.com looked like this at the time of the -above error: +This says to accept mail from that adelphia.net user despite the +SPF fail, but only after annoying them with a DSN about their ISP's broken +policy. + +If there is no match on the full sender, the domain is checked: <pre> -msg.euxiphipops.com TXT "v=spf1 mx ptr a include" +SPF-Neutral:aol.com REJECT </pre> -</td></tr> +This says to reject mail from AOL with an SPF result of neutral. +This means AOL users can't use their AOL address with another mail service +to send us mail. This is good because the other mail service is +likely a badly configured greeting card site or a virus. -<tr><th>TEMPERROR</th><td> -Temporary errors result in a 451 "Try again later" response. The sender -should retry the message at a later time. +Finally, a default policy for the result is checked. While there are program +defaults, you should have defaults in the access database for SPF results: <pre> -2005Jul24 07:33:13 [29846] mail from <quickenloans@rate.quicken.com> ('SIZE=73775', 'BODY=8BITMIME') -2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout +SPF-Neutral: CBV +SPF-Softfail: DSN +SPF-PermError: DSN +SPF-TempError: REJECT +SPF-None: REJECT +SPF-Fail: REJECT +SPF-Pass: OK </pre> -</td></tr> -</table> +<h2> Reputation </h2> + +If the sender has not been rejected by this point, and if a GOSSiP server is +configured, we consult GOSSiP for the reputation score of the sender and +SPF result. The score is a number from -100 to 100 with a confidence +percentage from 0 to 100. A really bad reputation (less than -50 with +confidence greater than 3) is rejected. Note that the reputation is tracked +independently for each SPF result and sender combination. So aol.com:neutral +might have a really bad reputation, while aol.com:pass would be ok. +Furthermore, when a sender finally publishes an SPF policy and starts +getting SPF pass, their reputation is effectively reset.