Nearly all websites these days send email and because of this the majority of developers assume that they “know how to send email from a website”. They continue under this assumption until they have a site or server of their get black listed by a Spam blacklist. Then they are forced to scratch their heads to try and figure out why this happened. Before you hit send on that email requesting to be removed from that Spam blacklist, let’s recap what you should be doing to make sure it doesn’t happen again.
So you’re developing a website, and you or your client wants this website to send an email.
Maybe it’s to visitors who sign up for your service, or maybe you have a “Send to a friend” service that allows your visitors to spread the word about your content or service.
Maybe you just want to send them a notification every time one of their friends virtually pokes them from your soon-to-be social network project (watch out for those Winklevoss twins).
No matter the usage, these added functionalities to your website all depend on E-mail and the assumption that these emails will get to their intended recipient.
Sadly, decades of abuse of the very E-mail services you are trying to deliver your message over by nefarious characters around the world has lead to us all being in a heightened state of Email DefCon Level 1. This means that at nearly every email gateway server around the world, system administrators and network engineers are working hard to make sure that Spam (or perceived Spam) doesn’t get through to their users’ mailboxes.
This means that you and your website need to be careful to not make the Uncle Sam’s of the network administration world go nuclear on your message.
A look at ways Spam filters can work
NOTE: Every different Spam filter on the planet does things a little differently (otherwise they’d have nothing to compete over), and I am not making blanket statements that this is how they all work. More simply, these are some of the things that Spam filters look at so that, in sharing them you as a developer can be more wary of them in your next project.
In order to make it easier to understand what I am about to explain I have going to create a virtual email for us to talk about. This example email’s basic information is:
Sender: JohnDoe@virtualcorp.com
Recipient: MarthaNoName@centuryco.com
Sending Servers IP: 202.202.202.1
Subject: Welcome to Widgets.com
Most Spam filters work under the pretence of incoming mail being given a score based on its risk of being Spam. They will do a number of checks and give your message a score – if this score is above/below the configured level for flagging your message as Spam or not it will then either let your email through to the recipients mailbox or reject it (depending on the Spam filter software being used, the response may be that your mail simply gets placed in a mailbox folder for review; think of GMAILs [Spam] folder).
Lets take a look at the lifecycle and some of the checks that occur, and then take a look at ways you can make sure your message makes the cut.
Step 1 – Your Email is received
The first thing the Spam filters usually do is take note of the domain name of the email’s sender (in our example email - virtualcorp.com) and the IP address of the sending server (in our example - 202.202.202.1) that it was received from as well as the domain name your server sends in its HELO command at time of connection.
Step 2 – Email interrogation
Now that a number of properties of your email have been noted, the Spam filter will then run a number of checks against your email. In the case of the bad guys there a usually a whole heap of things that happen outside of the list i am about to explain; such as heuristics on the email’s body content. As we are sending normal e-mail all we should worry about are the flags that could cause a false positive to occur.
DNS Background Check
The Spam filter will probably then do a reverse DNS lookup to see if the IP address of the sending server resolves to the domain name that is used when initiating connection (The HELO message) or the domain name of the email sender (in our example - virtualcorp.com).
Is this the reverse DNS hostname used in the server’s connection HELO message? Is this reverse DNS hostname included in the SPF records for the sending domains DNS? (shown later in this post). Is this server even connected to JohnDoe@virtualcorp.com?
It may then look up the Forward DNS entries for the domain name of the email’s sender (again this is virtualcorp.com) and look to see if the A record for that domain points to the IP address of the sending server or an IP in the same subnet block (i.e is the sending server actually the webserver that hosts the site or connected to this site in any way?).
All of these will be added to the email’s risk score. Some filters rate some checks higher than others.
Reverse DNS How-to
If you want to look up the reverse DNS entry for the your server’s IP address the simplest way is to visit a site like this one and enter your servers IP address.
Alternatively, open a command prompt (if using windows) or a terminal (if using a Unix flavour) and type the following:
Nslookup [IP ADDRESS YOU WANT TO LOOKUP] Hit Enter
The result when looking up the reverse DNS entry for Google’s DNS server 8.8.8.8
DNS SPF Record Check
Another common approach a Spam filter will take is to look up the SPF record and/or the Sender ID SPF record for the domain name you have as sender in your email message (again this is virtualcorp.com).
DNS SPF records define a list of servers that are allowed to send email on behalf of a domain name. They are an important way of defining servers that are not normally associated with your domain name to be recognised as being verified – if you are using a server that is shared across your webhosting ISP’s other customers, this allows you to white-list this server as being related to your website’s domain (a connection that normally might not be a clear as it should be).
Example
The following SPF record for example.com indicates that only one server can send on behalf of example.com and that is the server mail.example.com;
example.com. TXT "v=spf1 a:mail.example.com -all"
What does this mean for us web developers?
You may being thinking at this point:
Who cares about DNS entries and all this other bumf?
I can send email as another persons email address – I've done it before!
If you are sending email from your website, then you should care – not just because your message might not get through this once, but because a lot of Spam filters keep a score for sending servers and may either report back to a blacklist if they constantly see bad mail coming from it or may just reject all mail from you in the future. When you are sending email to a huge network of email servers (such as Gmail or Hotmail), them remembering you as sending bad mail is not a good thing.
This means that by not playing by the rules, you can actually over time get your server blacklisted.
Additionally, just because it hasn’t happened yet, doesn’t mean it won’t happen one day.
So what can be done to try and allow our E-mails get through?
After all this information, it might not be 100% clear what you can do to make sure you E-mail gets to its intended recipients and that you don’t set off any Spam false positives in the process. The list is quite small, and simple and usually if you stick to a few simple rules you should be fine.
Lets take a look at how you can use this information to your benefit.
Never send email from a sender address on a domain you do not control
If you are building a website that sends email, try to avoid sending as another user.
This means if you are building a “send-to-a-friend” form, or a “send this user an email” functionality always try and send from an email address on your domain, not as the user’s email address – you can add the user you are sending-on-behalf-of as the “reply-to” email address, but always send from a sender email address that lives on a domain you control.
If your websites domain name is “widgetsareus.com” make sure you send from an email address such as “website@widgetsareus.com” not “ladiesman69@hotmail.com”
Make sure that you have your DNS ducks are in a row
If you are sending from an email address on a domain that is hosted on another server (such as using a shared SMTP server from your webhost), get your DNS in order. If you are not in control of your site’s DNS make sure that whoever is knows about the following;
- The ISP that hosts your server sets up a valid reverse DNS entry for your server’s external facing IP address that it points at a hostname that is a sub-domain of the domain name you wish to send as if possible. If you are using a shared SMTP server this is usually not possible as the server will be sending email for many different domain names not just yours – in this case SPF records are even more important.
- Setup an SPF and Sender-ID SPF record for your domain name that includes the external facing IP address or hostname of your mail server. If you need some help, give the following sites a visit:
- SPF Record Documentation
http://www.openspf.org/SPF_Record_Syntax - Sender ID Setup Wizard
http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/
- SPF Record Documentation
Verify your SMTP server is configured correctly
If you control the SMTP server that is sending your mail make sure it is configured to give its reverse DNS hostname in its HELO connection greeting from its reverse DNS hostname or the domain name that you are sending from.
Also make sure that relaying is turned off – THIS IS PRETTY CRITICAL.
To check if you have this turned on, check out an open relaying tool such as the one of the following:
http://www.mxtoolbox.com/diagnostic.aspx
Verify that your SMTP server is not on any blacklists
There are hundreds of Spam blacklists, but MXToolbox has a great tool for checking your servers IP address @ http://www.mxtoolbox.com/blacklists.aspx. If you find yourself on any of the above, double check that you have configured your server correctly and politely email the administrators of the blacklist and request to have your server’s IP address removed.
Summary
Sending E-mail from your website is not a complicated task (as your probably already know). Trying to assure that your email messages get to their recipients is not too hard as long as you stick to a couple of simple rules.
Hopefully the list above, will allow you to get back to doing what you do best – rocking the web.
I have written a follow up post on using DKIM and Domain Keys to “sign your email” when you want to take this a step further.
Check it out:
When you really need E-mail delivered – Signing your mail using DKIM.