Revolution Systems Blog

Email, Templates, and Perl

I have been meaning to talk about one of my new favorite Perl modules, MIME::Lite::TT::HTML , for quite a while now.  As I mentioned in a previous post, there are a bazillion different ways to send an Email message from Perl.  This one is just my new favorite.

Here is a short list as to why:

• Can be used for complex multi-part messages and handles attachments easily
• Built upon the equally great MIME::Lite module
• Allows you to easily template your messages using the familiar Template Toolkit package

The templating part is, in my opinion, the important part.  How many times have you had to go edit some source code just to change the text or subject of a message?  Isn't that just terribly annoying. We use configuration files, MVC with HTML templates, etc, etc. to not hard code things into our apps, but for some reason many people ( myself included for years ) have neglected Email.

Not any longer, I've switched to using this module as my standard way of sending Email these days.  If you are interested in learning more about MIME::Lite::TT::HTML, check out my short howto Sending Email with Perl Best Practice on the subject.

SMTP Connections: How to handle large loads

I recently came across several blog posts about the declining state of E-mail due to spam.  Specifically these:

• Spam Volumes at accidental-DOS Levels
• Another War We Aren't Winning: Us vs Spam
• E-mail as we know it is doomed

I've been running E-mail servers for myself and others for over 10 years now and I have to agree that with the current version of SMTP we all use, there isn't much that can be done about spam that isn't already being done.  If you've got some RBLs, SPF, anti-virus, and a decent spam filter setup, there isn't much more you can do.  Sure you can get +/- another percentage point, but you won't really find a solution that is 100% effective 100% of the time.  It just isn't possible with the current standards.

However, these articles also discuss another issue that is overlooked by everyone who doesn't run a large E-mail system.  By large I'll say over 1,000 users. Around that point you start to run into problems being able to handle the shear number of incoming SMTP connections.  Note that you may hit this point long before 1,000 users depending on hardware and your personal traffic levels. But I digress.

I haven't found any Open Source solution to this problem, but I've come pretty close.  TrafficControl is a commercial product from MailChannels that is built upon the good Open Source base of Apache and mod_perl. TrafficControl does two things for you:

• It uses async I/O to proxy incoming SMTP connections.  This has allowed me to handle 10x the number of connections on the exact same hardware
• Allows you to configure throttling rules based on RBLs, Operating System, etc. and choke off the bandwidth to suspicious sending servers.  This reduces a great deal of spam as most bots simply move on if they are not getting go flow.  After a configurable time period it will remove the choke hold and allow the message to continue on normally.  Servers that follow the SMTP RFC have no problem with this as they hang on during the process.

I've been using it for about a year now and couldn't imagine trying to run a large E-mail system without it now.  I encourage you to check it out if you see you are having a similar problem.

greet_pause -- A new anti-spam feature in Sendmail 8.13.x

With the release of Sendmail 8.13.x there is a new anti-spam feature that you should seriously consider taking advantage of. The new feature is called greet_pause.

During an SMTP session, after the sender connects to port 25 on the recipient's MX, it should wait to receive the Sendmail banner prior to sending any further data.  In the interest of spewing as much spam as possible, many spammers disregard this and simply force send all of the SMTP commands at once and move on.  The receiving MTA typically takes this spew and delivers it.

greet_pause tells Sendmail to wait for a specified amount of time before sending the banner and if the sender spews commands early the message is rejected.  This can drastically cut down on the amount of spam you receive on a Sendmail system.  You implement this by adding the following to your sendmail.mc:

FEATURE(`greet_pause',5000)

This tells sendmail to wait 5 seconds before displaying the banner.  This should be plenty of time for a spammer to begin spewing, but short enough to not cause any timeout problems for legitimate E-mail.  You can set it to whatever you think is best, I would suggest something in the 2-10 second range.

Some legit MTAs might still have problems with this, so you can whitelist them by adding the following to your acess file and rebuilding the database:

GreetPause:localhost    0
GreetPause:24.124.0.1   0

This would not delay the banner for localhost and the IP 24.124.0.1. I recommend at least putting in localhost as there is no need to delay yourself. 

Sendmail Virtual User Trick

One of the things people find difficult about Sendmail is virtual users.  These are defined in the virtusertable file ( usually in /etc/mail/virtusertable ). This file instructs Sendmail to translate a "virtual" user into a real user or alias. The reason I mention aliases here is because, with Sendmail, you can have a virtual user that translates into a alias for multiple local and/or remote E-mail accounts.

A situation some people run into is that they want all usernames at domain2.com to be delivered to the same username at domain1.com, except for a few users.... If you really wanted all users to map to the other domain it would be as simple as adding the domain into the local-host-names.  It is the need to have *most* users map to the other domain where the virtusertable file comes in handy.  For this example, let's assume that we want all users names @domain1.com except for postmaster, webmaster, and support to map to the same username @domain2.com this would be accomplished by adding domain2.com to the local-host-names file to tell Sendmail that we wish to receive mail for that domain and then adding the following to the virtusertable file:

postmaster@domain2.com: mailadmin@example.com
webmaster@domain2.com: webmsater@example.com
support@domain2.com: support@example.com
@domain2.com: %1@domain1.com

The first three lines tells Sendmail to send messages sent to those three usernames at domain2.com to the appropriate remote E-mail addresses. The last line instructs sendmail to send *any* other usernames sent to domain2.com to the same username at domain1.com. Note that this will also include any fake usernames that a spammer might send to. The E-mail server at domain1.com will still be responsible for determining what is or is not a valid username. After you've added those entries to the virtusertable file all you need to do is rebuild it and it becomes active.