August 2004 Archives

Useful little link: Vim Commands Cheat Sheet

Julian released 4.33.3 this morning. It's basically a stable version based on the beta released a couple of days ago. He also released some SpamAssassin v3 related scripts yesterday, but they seem to have been removed from the downloads list (or maybe I'm blind)

Came across a very interesting entry on Justin Mason's blog: Open Source v Closed Source spam filtering which explains how spammers test closed-source email filters. I suppose it makes sense, but it's still quite scary that theyse people invest so much time and effort in circumventing company's best efforts to protect their clients' inboxes.

Julian released two beta versions of MailScanner today - 4.33.1 was quickly replaced by 4.33.2 The more interesting changes are in MCP, which is now fully integrated into MailScanner.conf. Although I'm not using it on any of our production systems it's still a very interesting feature that could prove useful in some situations. Another feature that was added not so long ago is the -v switch (case insensitive): MailScanner -v will print out information on the host system, including the OS, perl version, version of MailScanner installed, as well all the required modules and any optional ones. This could be very useful when deploying/updating on multiple machines and for debugging purposes. One of the areas that is being actively addressed is support for SpamAssassin 3. Another release candidate came out a couple of days ago, so hopefully support for autolearn and other functions will be working properly by the time it comes out. I installed the latest beta on one system just to see how it is before pushing it out onto any important servers. As was to be expected the install and upgrade was easy. A couple of people had reported some performance problems with a couple of the earlier releases, but watching our mail logs I didn't see anything out of place.

Firefox is a very nice browser and makes an excellent replacement for Internet Explorer. One of the things that was a little annoying after installing Evolution in Debian was that hyperlinks in emails did not work. Niall provides a simple way of fixing this here.

As I keep on losing the links I'll post them here for myself and anyone who needs them: Full list i386

Wired networks are very restrictive, but you really don't appreciate how much freedom wireless can bring until you actually have it. A couple of months ago we picked up a few wireless pcmcia cards on eBay. They are Cisco aironets which work perfectly in linux. You simply pop it in and it works (unless you're using Suse, which is a little demented when it comes to network interfaces) immediately: The only thing that remained "wired" was my desktop, so a PCI card was needed. I finally picked one up on ebay this evening, so I hope to have it in a few days: Once I've got this installed I'll finally be able to banish our rather noisy router (running pebble linux) to another room!! :mrgreen:

This server is now ipv6 enabled! ipv6 info has a lot of the heavier technical info. As this server is running Apache2, which has native support for both ipv4 and ipv6, enabling it was simply a matter of changing the vhost config slightly. The pop/imap server, dovecot, supports ipv6 as well. All you need to do is make a couple of minor changes to the main configuration file. Unfortunately our ISP does not offer ipv6 yet, so we use a tunnel over ipv4, while our ipv6 allocation is tunnelled from Esat. Probably not the cleanest way to do things, but it works.

What would you consider to be essential tools for your PDA? The standard applications that ship with it leave a lot to be desired. Obvious ones: SSH client: The only one I've found that is anyway usable is PSSH. It works fine, but the default display setting will make you go blind, so change them as soon as you can. Java: You can grab a Palm specific version over at PalmOne. When I first got a copy of it back in December 2003 there was very little information on there, but they seem to have taken some more interest in it recently, and now include links to a number of sites where you can download midlets. Email/Web: I'm still trying to find a usable IMAP client. I tried the Qualcomm suite, but it didn't handle my IMAP folders very well. Palm have an email client, but it's not bundled with the OS or available for free. I'll keep rummaging........

I get very frustrated sometimes with release dates for films. You see the publicity stuff on imdb, but you've no idea when they are going to be released in Ireland. This site seems to have all the answers.

I got a Palm Tungsten T2 a few months ago, which is very handy if I'm on the move. It has bluetooth support, so I can easily pair it with my Nokia and connect over GPRS. One of the more useful things that I could use the Palm for would be SSH access, but trying to type commands on the Palm's builtin "virtual" keyboard is far too frustrating. The solution was to get a portable keyboard: Although it folds up to approximately the same size as the Palm itself this is a full-size keyboard, with all the "bells and whistles". Time to find more apps for my Palm :mrgreen:

Ross pasted a link to me this morning which had me in stitches: Spammers Sending Messages from the Future

I just found a link to the mail archives: Chkrootkit Mail Archives

Niall managed to get Zempt working with Wordpress. Handy!

Although the MailScanner mailing lists can be a wonderful resource it has become a victim of its own success. Over the last couple of months the signal to noise ratio seems to have changed dramatically. A lot of queries posted to the list could be easily resolved if people read the FAQ, MAQ or manual or even the comments in MailScanner.conf How hard can it be?

Just reading a few of the lists this morning and noticed the usual problems with using an RBL to block mail at the MTA level(name removed to protect the original poster's identity): "But the problem is, some of my users also are unable to send their emails using SMTP server as their "dynamic" IP is banned because some of the ips are listed in spamhaus. They keep getting the error above. How can I rectify this? Is there a command for me to add to allow user based on their IP address or email address? " Solution available: none if you insist on using spamhaus to block mail at the MTA level. Denying access to your MTA based on RBLs is demented and wrong. Why? Because you cannot rely 100% on an RBL's accuracy. Does this mean that RBLs are inaccurate? No, of course not. You just need to understand how they work and how to use them. If you score against an RBL you will get the right results, as you will score based on a number of criteria ie. there isn't a "single point of failure" The root of the problem does not lie with the RBL maintainers, some of them even state on their respective sites that blocking is a bad idea, but with misinformed sysadmins. If you are running a mail server for personal use you can do pretty much what you like, as you are the only person who is going to suffer if/when things go wrong. However if you start implementing blocking in a business environment you are simply asking for trouble. Of course you are going to see a noticeable reduction in spam, simply because you'll have blocked a large portion of the internet. Spamhaus is a fantastic resource and can help to significantly reduce the amount of spam arriving in your users' mailboxes, but it is not a good idea to block all mail emanating from IP ranges listed by it. Some discussion recently on the SURBL list has centred around the length of time an IP is listed in Spamhaus. Although it makes interesting reading from a theoretical point of view, its practical implications are not going to bring any significant change to usage. The idea that an IP may be listed for a brief period and then delisted as the issue is addressed is not unique to Spamhaus. In reality the only thing that matters is whether the IP is listed at the time of arrival on your scanning server ie. whether it will be flagged or not.

A couple of people were asking me where they could find rpms for SA 2.64, so here's a link to help you: DAG rpm archive Personally I prefer to do it from cpan or source, as the rpms have a "charming" tendency of installing all sorts of things that I really do not want.

Background We (Blacknight Solutions) have been offering email filtering to our clients since early 2002. We first began "experimenting" with spam filtering as we saw that the problem of spam/uce was growing exponentially and neither we nor our clients wanted to have our inboxes taken over by rubbish. For the first 10-12 months after implementing server-side filtering we did not block email, as we preferred to merely tag it and deliver it. By tagging the subject line of emails in a consistent manner our clients were able to filter potential spam into another "folder" for examination. After our initial tagging period, which involved constant tweaking of the scoring criteria, we moved from tagging to storing. Currently we offer email filtering at different levels to our clients. At the lower end of the scale the clients' email is scanned and stored by us without any user intervention ie. no customised black/white listing etc., while at the higher end customisable rules and criteria are implemented. Scope and motivation of this article Over the past 6 to 12 months the subject of email filtering has begun to attract more publicity both in "techie" circles and amongst the general public. One of the reasons for writing this article is to address some of the common misconceptions about email filtering and best practices. After following many of the discussions on technical mailing lists and bulletin boards over the last few months the author feels strongly that some people's approach to email filtering is both misinformed and dangerous. Due to the scope of the subject matter this article will probably be split into a number of shorter articles ie. parts, but comments from readers will be welcomed. This article will address some of the issues involved in implementing email filtering for business and discuss some of the methods currently being used both in industry in general and by the author. Due to the nature of our service the finer details of our setup will not be revealed, but general criteria and methodology will be discussed. Any opinions expressed in this article are the author's and are based on the author's experiences. Definitions In order to avoid confusion a number of terms should be defined for the purposes of this article. UCE: unsolicited commercial email For many people there is no clear difference between the two. However a number of things may give some indication. If the sender of the email makes it clear where they obtained the email address and how you may be removed from the list it is helpful, although there is a very valid argument about unsubscribing from lists to which one was never subscribed. Why should the onus be on the recipient? It also informs the sender that the email address is valid. In my case I can usually tell if an email address has been scraped or not based purely on the address. A number of my older email aliases have not been used for at least two years due to the volume of spam that they were receiving. As a result I can safely say that any mail received to info@ is spam, as the address has not appeared on our website for at least two years, nor have I used it for at least that period. This is not a matter of a spam trap but more a simple case of applied logic. The only way you could get that address is through a spammers' database. spam: If you look at the variety of definitions offered by Google for this term you should immediately see part of the problem. Depending on who you talk to scope of the definition can change quite dramatically. In simplest terms it may be best to refer to "spam" as unwanted commercial email ie. mail being sent on bulk offering you commercial services that you do not want. Even that definition is not very clear, but it may help as a starting point. The type of spam that causes most problems for business is adult in nature and may vary from the extreme hardcore porn variety through to the adverts for sexual aids both herbal, chemical and physical. Tools There are an ever increasing number of tools and services available to help you block spam/uce available on the market. These can be divided into two groups: client-side: The software resides on the user's pc. It may be an independent piece of software or an addon to an email client. For example email clients such as Outlook 2003 and Eudora include spam filtering tools. Although client-side tools have their merits they do not address the primary issue with spam, which is the cost in both time and resources in downloading unwanted email. For this reason I believe that we should focus on server-side solutions. Another issue with client-side applications is that they do not update often enough, so they cannot address the issues that each new wave of spam brings. Server-side: As the name suggests these are tools that work directly on the mail server. The advantages to using server-side tools are numerous. By blocking/filtering mail on the server you move the administrative responsibility away from the user to the server admin and their choice of tools. ISPs and hosting companies' mail servers are connected to the 'net 24/7 via high bandwidth connections, so although the level of unwanted email may incur a varying level of resource usage at the server level this will have significantly less impact than the resource usage at the client level. Unlike client-side tools those used server-side have the ability to update not only in realtime but also through collaboration with other servers and through the usage patterns of the users being served. Common Problems and misconceptions There are a number of problems facing any provider of email filtering.

• Technology
• Client expectations
• Accuracy
• Contractual issues

Technology affects both the tools being used to stop the spam and the those being used by the spammers to send the email. Both are in constant evolution. A number of examples spring to mind: Habeas: Users of habeas' system are allowed to include a number of lines in their outgoing email which shows that they are valid email users. A number of spam blocking tools, such as SpamAssassin, allocate a negative score to emails with this header. Unfortunately spammers became aware of this "hole" and started using it as a way of getting mail past people's filters. The only viable solution was to adjust the negative scoring assigned to the habeas headers to compensate. Bayesian filters: Believed by many to be one of the most powerful weapons in the spam fighter's arsenal Bayesian filters score mail based on the frequency of words in ham and spam: The Bayesian classifier in Spamassassin tries to identify spam by looking at what are called tokens; words or short character sequences that are commonly found in spam or ham. If I've handed 100 messages to sa-learn that have the phrase penis enlargement and told it that those are all spam, when the 101st message comes in with the words penis and enlargment, the Bayesian classifier will be pretty sure that the new message is spam and will increase the spam score of that message. (taken from http://wiki.apache.org/spamassassin/BayesInSpamAssassin) Needless to say spammers realised this and began to use a counter technique to mess up the Bayes databases (Bayes poisoning) Other issues that we have identified are simply related to the use of outdated tools. Older versions of spam assassin, for example, will falsely score email from Outlook 2003 as spam, as the outlook version string appended to the header is not known to it. Although this issue is easily addressed via an upgrade to version 2.6* or later many corporate users relay on 3rd party vendors for email filtering. If the 3rd party in question is negligent email may be lost. Client Expectations are, in many cases, one of the biggest issues. Many clients expect a spam filter to be 100% effective. This is not possible and anybody who says otherwises is either naive or foolish. No matter what technology you use to filter mail you will always have to balance the likelihood of getting false positives (ham marked as spam) versus false negatives (spam marked as ham). If you wish to reduce the level of false negatives to zero you will get false positives. Why? There are a number of reasons for this, including badly formed mail, blacklisted netblocks or spam like content, to name but a few. A more pragmatic approach is to push the boundaries as far as possible in order to minimise the risk of false positives. If you approach it in the other direction you run the risk of losing valid email. The loss of valid email when you are filtering mail in a business environment is simply unacceptable. Although those in the industry are aware of the inherent unreliability of email as a communication method end users expect it to simply work. As many businesses rely on email as one of the primary forms of communication with their suppliers and customer base any delays or problems can have economic consequences. Tools of the trade MailScanner Mailscanner is an award winning mail filtering/scanning package which is capable of complex scanning of email, both inbound and outbound. The default configuration is sane, however we have extended and customised ours extensively allowing us to implement email filtering to our taste. MailScanner makes use of the Spam Assassin libraries but it does NOT use spamc or spamd. Spam Assassin Current version available is 2.64, although most people would be using 2.63 as the 2.64 release was only a couple of days ago. Version 3 is currently in development and should be released within the next few weeks. The 3 series will bring a number of radical changes and improvements to the engine, but as it is still beta / release candidate it is not being used on many production systems. The documentation available on the SA website is comprehensive and covers installation and configuration for both server-wide and per user installation. Blacklists RBLs Often referred to as blacklists, RBLs, realtime blacklists or DNS blacklists. These can be used directly at the MTA level or via MailScanner, spam asssassin or similar. The problem with RBLs is that they change all the time. Of course this is the primary reason why people use them, but if you opt to block based on RBLs you are asking for trouble. One of the most common problems I have seen with users of RBLs stems from either ignorance or admins being naive. If you are going to implement an RBL you should know what it does and why it does it. If I use an RBL I have a reason for doing so, but that may not suit your particular environment and vice-versa. Before you add an RBL into your mix you should take the time to visit the RBL's homepage and read up a little. Find out what the listing and delisting criteria are. If they seem sane then you may choose to use it, if you have any doubts then don't use it. Unfortunately a lot of the spam organisations choose to target RBLs either via DDOS or legally. If an RBL becomes inactive then its database is no longer of any use to you, or it may even damage your ability to filter mail, as was the case when a well-known RBL decided to blacklist all mail. RFC Ignorant, for example, is not a good choice as they have blacklisted the entire IE ccTLD as well as any domain that does not meet their criteria. Other RBLs, such as spamhaus, use quite sane criteria, however due to ISPs inaction large blocks of innocent Ips may be listed. Simply put: If you choose to score based on RBLs you will see good results. If you choose to block based on them you are shooting yourself in the foot.

My new desktop machine, which I got a couple of months ago, came with an onboard graphics card. The card was "ok", but as I like playing some games, watching DVDs and doing other stuff which is fairly intensive I decided to upgrade. My choice of graphics card was very restricted as the PC, a Fujitsu Siemens, does not have an AGP slot. Why? I have no idea, but considering that I got the PC at a very good price I'm not going to complain... well not too loudly anyway. Installing the new PCI card was not too hard, although the case of the PC is rather packed, so you feel as if you need to have long narrow fingers in order to access certain areas. Getting the card up and running was easy enough, except I forgot to disable the onboard card via the BIOS the first time I booted! The card came with the windows drivers, so that wasn't an issue. However when I wanted to get a dualboot systen running I ran into problems. Fedora Core 2 didn't have any issues in recognising the card and getting it to work in 2d. Debian, however, simply refused. Although I had the assistance of Niall, who knows Debian inside out, we couldn't get it to work at all. I finally got it working this morning thanks to the invaluable help of the #linuxhelp channel on undernet. The solution was actually quite easy - as long as you knew it. Running the command: dpkg-reconfigure -plow xserver-xfree86 brings up the X windows configuration utility. Debian unstable, which I am now running on my desktop, recognised ATI as being the card vendor I wanted to use, probably because I set that the last time I tried to set it up during the install. The main problem was that the configuration tool was asking for the PCI bus code for the graphics card. I know very little about setting up X and usually just want these things to just work. I normally do not have the time nor the patience or even the know-how to go messing about with config files on the command line. The solution was to run the lspci tool, which basically probes the pci bus and gives an output of what it finds. For some insane reason running: lspci from the prompt didn't work, probably because it wasn't in my path. The general consensus was that it should have been installed, so after a bit of digging I found it and was able to run the command as: /usr/bin/lspci That outputs a lot of text to the screen, so I got it to write to a text file: /usr/bin/lspci > /home/blacknight/lspcioutput which I could then read using less. The relevant line was there: 0000:02:09.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 SE] (rev 01) So i put : 02:09.0 into the relevant box in the x config utility, followed it the rest of the way through and now have X with KDE running nicely. Admittedly i haven't tried logging in and out yet, so it may die again, but at least it worked once :mrgreen:

According an article published on SMAU.it open source software drives the backbone of the 'net. Although this is not exactly "groundbreaking" news it is nice to see more international press coverage of open source. SMAU is well worth a visit if you can make it to Milan in the autumn and completely dwarfs the Irish ICT Expo.

Our google ranking has magically improved almost overnight. I'm not going to complain about that :mrgreen: , I just wish I knew why our site sometimes vanishs completely. Interestingly enough Google has now learnt about our domain aliases, so all the results point to one domain, which makes more sense. We aren't being penalised as we don't use doorway pages or any other "tricks". It has also "learnt" the relationship between me and the company site, possibly due to my email signature appearing in indexed mailing lists. I also came across this funky little tool that supposedly outputs the PR of a page: I'm not sure how accurate or useful it is, but it's definitely interesting. If you look at this entry on the site's main page it reflects the blog's PR, while the entry on its own doesn't seem to have registered with Google just yet, which is hardly suprising.   Google Hacks Exposed: Improving Your Rank on Google

Tom has started a campaign to force Beamish to do something about their buildings. The state some of those buildings are in is truly shameful, expecially when you consider that Cork is going to be European City of Culture in 2005.

I orginally setup this blog as a number of our clients were asking about running blogs on our servers and I hadn't any experience with the various tools involved. After trying a few different perl and php blogging tools I settled on Movable Type.