Spam Filtering: January 2005 Archives

In what could best be described as "madness" US ISP Verizon has decided to block all of Europe to stop spam. Full article available here Thanks Verizon!

If you are going to run a mailing list you need to be very conscious of the problems you face. At the technical/design level you need to send emails that people will be able to read. Plain text emails are obviously going to be legible in all email clients, but are not as attractive as HTML ones. Unfortunately no two email clients are going to render your emails in exactly the same way. You need to check against the most common email clients and webmail interfaces to get a "feel" for how the final product is going to look. Bear in mind that just because an email client can display HTML does not mean that you should try to use the same methods as you would when designing a web page. Do not include any client-side scripting (javascript etc) as this will be blocked by any good system. Do not include forms, as these will probably be stripped out. If you ensure that your HTML is valid you are less likely to find your email being blocked by spam filters. If you are sending from your desktop make sure that your PC's date and time are correct. If you are sending from a server make sure that your mails are being sent out by a user ie. Not being sent by nobody@ or apache@ If sending from a server make sure it has reverse DNS setup - lack of a reverse can lead to email being blocked Make sure that you keep your list clean ie. That you do not generate large numbers of bounces Make sure that your list conforms to permission based standards: "The difference between senders of legitimate bulk email and spammers couldn't be clearer, the legitimate bulk email sender has verifiable permission from the recipients before sending, the spammer does not." Putting a simple footer in each email explaining not only how to unsubscribe but where the person subscribed from will help avoid issues Including unsubscribe instructions is NOT enough Subscribing people to your list without their permission is NOT acceptable Sending mail to people/companies because you feel that they "may be interested" in your products or services is NOT acceptable If you have an existing relationship with clients / suppliers or potential clients you may send them email, however you should always make it clear how you got their email address if you do not use some form of subscription confirmation. If you get reported for spamming it can ruin your business. References: http://www.spamhaus.org/mailinglists.html http://www.spamhaus.org/definition.html http://www.georgedillon.com/web/html_email_is_evil.shtml

For the last couple of months we have been tracking our logs using Vispan with the GeoIP. We can easily see where viruses and spam are coming from and thus draw up a top list of countries. For example, so far this month the top sources of spam have been:

• United States
• Korea, Republic of
• China
• Ireland
• United Kingdom

NB A percentage of these emails would have been tagged as spam even though they are actually viruses as some of the DNSBLs and rulesets will pick up on viral emails A simple solution to the spams from Asia would be to block those countries completely, however that would cause issues as we have clients who trade actively with those countries. A better solution is to apply some form of weighting to email from certain countries. We are based in Ireland, so a large proportion of our clients and their contacts are too, so assigning a certain degree of "trust" to Irish IP space would seem to be a logical step. This does not mean that email from Irish IPs is trusted, however we are going to assume that the likelihood of spam coming from an Irish IP is slightly lower than from a non-Irish IP. We then take the countries that are most problematic, such as Korea and Hong Kong. All email from those countries is considered untrustworthy and weighted appropriately. The rest of the world is treated equally. The results of this scoring can be best illustrated by looking at a couple of border line cases: Jan 14 22:36:54 av MailScanner[29107]: Message j0EMandY027564 from xxx.xxx.xxx.xxx(xxxxx@xxxxx.ie) to xxxxx.ie is not spam , SpamAssassin (score=7.097, required 7.4, HTML_70_80 0.51, HTML_FONTCOLOR_MAGENTA 1.00, HTML_FONTCOLOR_UNSAFE 1.00, HTML_FONTCOLOR_YELLOW 1.00, HTML_MESSAGE 0.10, HTML_MIME_NO_HTML_TAG 1.00, HTML_SHOUTING4 0.50, HTML_TAG_BALANCE_BODY 0.18, HTML_TAG_BALANCE_TABLE 0.19, HTML_TAG_EXISTS_TBODY 0.10, MIME_BASE64_NO_NAME 1.00, MIME_HTML_ONLY 0.10, RCVD_IN_NERDS_IE -2.00, SARE_HTML_NO_BODY1 1.03, SARE_URI_DIET 1.37) The header above is from a legitimate health newsletter. The Irish IP that sent it was assigned a negative score (-2) which brought an otherwise false positive safely down below the limit. On the other side we have an email that would otherwise have got through: Jan 15 13:39:26 av MailScanner[3774]: Message j0FDcoF8019064 from xxx.xxx.xxx.xxx(xxxx@xxxxx.com) to xxxxx.com is spam, SpamAssassin (score=9.663, required 7.4, RAZOR2_CF_RANGE_51_100 1.00, RAZOR2_CHECK 2.06, RCVD_IN_NERDS_CN 3.50, RCVD_IN_SORBS_DUL 1.00, WS_URI_RBL 2.10) The email scored against two DNSBLs, but would have only scored 6.1, however the sending IP (from China) pushed it nicely over the limit. How are the scores assigned? We use a custom ruleset that polls the IP list provided by http://countries.nerd.dk/ and assigns a score based on the result. An example score is below: header __RCVD_IN_NERDS eval:check_rbl('nerds','zz.countries.nerd.dk.') describe __RCVD_IN_NERDS Received from a spam country tflags __RCVD_IN_NERDS net header RCVD_IN_NERDS_AR eval:check_rbl_sub('nerds','127.0.0.32') describe RCVD_IN_NERDS_AR Received from Argentina tflags RCVD_IN_NERDS_AR net score RCVD_IN_NERDS_AR 2.5 Thanks to a recent thread on the spamassassin users' list SpamAssassin