Did you know that most of the email coming to UNC-Chapel Hill email addresses is junk? Spam, phishing, viruses and other bad emails account for 92.8 percent of the email received at Carolina over the previous six months. That is 10.7 million junk mail messages every day versus 709,000 valid email messages per day on average. If it were not for our email hygiene applications and devices, it would be nearly impossible to use email effectively.
In order to mount a strong defense on that incoming tidal wave of bad data, UNC-Chapel Hill uses vendor products that have two categories of techniques:
Content filtering Reputation filtering
FYI #1: The entire email is scanned
Content filtering uses proprietary algorithms developed by the vendor to scan entire email content to determine if a particular message is a problem. These content filters are extremely resource intensive as each message passing through must be thoroughly scanned and compared to tens of thousands of rules. While the content filters are very good, they would not be able to scan ALL inbound messages and deliver to the recipient in a timely fashion. It is for this reason that most vendors including Cisco rely on reputation filters to filter most of the mail and content filtering to filter out the more nuanced attacks, which are much smaller in volume.
FYI #2: The reputation of the domain and IP address matters
Reputation filtering is based on reports by applications and individual users and identifies whether specific IPs are sending problematic emails. Cisco runs SenderBase.org, which purports to be the world’s largest email and Web traffic monitoring network. The site maintains a SenderBase Reputation Score (SBRS) for each sending domain (like yahoo.com or unc.edu) and associated IP addresses. The score is based on reports from around the world on domains and IP addresses that are known senders of spam, viruses and other bad content.
The effectiveness of the reputation is filter is surprisingly good. Of the 10.7 million messages per day received at UNC-Chapel Hill, we filter out nearly 92 percent by reputation filtering alone. The remaining messages are then sent on the content filters for more intensive scanning.
FYI #3: Marketers’ spam creates ripple effect
We could not survive without reputation filtering, but occasionally “good” senders like UNC-Chapel Hill or other universities will have compromised accounts that send spam through their systems and then their reputation scores fall. This results in punitive measures being imposed on the sending organization, such as limiting the number of messages per hour a domain can send or even complete blocking of email.
While both types of filtering are effective and reputations go up and down resulting in blocked, rate limited or full flowing email, there are other issues that further complicate the email hygiene landscape. The most challenging Catch-22 situations involve marketing houses like Constant Contact. We have University users of these services that want and need to send important messages to many of our UNC-Chapel Hill mail users. Some of these marketing houses also endeavor to send spam, not from Carolina, of course, but once a reputation score at an email marketer falls, all the users of that company suffer.
As a result, we get requests to “whitelist” these senders to allow them to send to UNC-Chapel Hill. In the past we have honored these requests, but the marketing houses (sometimes sent by their customers without their knowledge) tend to abuse the open filters and send spam into Carolina, causing complaints from our community.
FYI #4: It’s a balancing act sorting bad from good
There is constant tension between keeping a tidy email inbox for our users by filtering out the bad messages and the need to allow valid entities to communicate en masse with the UNC-Chapel Hill communities. Sometimes one community wants messages to get through that other communities may complain about — in the case of borderline spam or marketing email.
FYI #5: The University remains vigilant
The takeaway is that we strive to do the right thing for the whole of the University every day through continual reexamination of our effectiveness, constantly updated vendor algorithms, use of the latest reputation scores, and vigilance in stopping our own accounts from being compromised and endangering our reputation. Our hope is that our customers never even notice.
Tim McGuire, ITS
Tim McGuire, Director of ITS Campus Infrastructure Services at UNC-Chapel Hill, has worked in information technology for two decades. In this blog, he explains how ITS filters out the bad email messages and senders to enable a safe and effective University email system.