Cloudmark Authority 2.0 boasts serious technology upgrades as abusive messages, the company says, now exceed 90 percent of all mail messages.

by Alex Goldman ISP-Planet Managing Editor [February 1, 2007]

San Francisco-based Cloudmark has announced the latest version of its software, Cloudmark Authority 2.0. The company focuses on serving the largest ISPs in the world. It claims to serve six of the ten largest ISPs in the U.S. and eight of the ten largest ISPs in Japan.

Spam volume is at the point where even storing it temporarily costs the largest providers money. Recently, we spoke to Wayne Lewis, the CTO of everyone.net (see The Technology to Run a Massive Mail Operation). Lewis told us that he's focused on reducing the number of I/O operations performed on each message, to improve the efficiency of the entire e-mail infrastructure.

Cloudmark is also avoiding accessing the disk, but that's not its focus. Instead, it's working on increasing the number of scans in order to improve the accuracy of the scanning results. Unless you're as large as everyone.net, many of your messages will not be delivered in real time, and you therefore can afford a slight delay in delivery if doing so will allow you to provide greater accuracy.

That's because however good you are, it's never enough. Jamie deGuerre, vice president of technology services, explains, "we have always had the highest accuracy. When we deploy with customers who have been using another solution, at first, they rave about the new solution. But then they become accustomed to the new level of spam, and start to become frustrated with that. So it's a goal here at Cloudmark to get the accuracy as high as possible."

How it works
One important tool is something the company calls active rescanning. deGuerre points out that since users generally check e-mail only once every 12 or 24 hours, there's plenty of time to scan each message multiple times. Of course, since the whole system would slow down if it accessed a stored copy, the company's "efficient rescanning" handles fingerprints of the message that are stored in a cache.

"Traditionally, rescanning was too costly," says deGuerre. "The spam filtering component of the infrastructure is the most CPU-demanding. Solutions require a lot of processing for each e-mail because the number of rules or heuristics is high. Having to retrieve the whole e-mail takes time."

Instead, Cloudmark Authority 2.0 uses what deGuerre calls "Cloudmark's unique spam technology, what we call fingerprints." The system compares elements of the e-mail (Cloudmark calls the elements "fingerprints", others have different names for this, such as Spam DNA) against a database that is updated every 45 seconds.

The list of fingerprints is so efficient that adding the fingerprints to the message only marginally increases message size while obviating the need for message retrieval during scanning. "We can retrieve the headers or meta data only and pass them to our Authority 2.0 product," deGuerre says. "We do 800 e-mails per second whereas others can do less than 100. For example, we scan every URL in the message and generate a hash for it. We generate 5 to 12 fingerprints per message, each of which is 10 to 20 characters in length."

deGuerre says ISPs with residential ISPs can be especially confident it will protect them against outbreaks. "Let's say an attack takes 12 minutes to infect the world. If we catch it one hour later, we can do a re-scan, and catch it before the subscriber checks their e-mail (say we caught it during the sixth minute of the outbreak). With the re-scan, we catch the entire attack even though we may have missed all or part of it initially."

Cloudmark made a name for itself with a crowdsourcing solution in which users rated messages as spam and a message that achieved a sufficient number of poor ratings would be blocked. This is still a part of the solution, and Cloudmark has upgraded its reporting features.

ISPs that implement anti-spam solutions need to remind customers that they're working hard. "Customers often never feel like they're part of the solution," says deGuerre, "because they never see it working."

"There's so much press about how spam is winning," says Dave Champine, Cloudmark senior director of product marketing. "There's an additional benefit here that's all about perception and the user experience."

Cloudmark calls another new feature secure unsubscribe, which is for handling a category of messages that some users perceive as spam but which other users want to receive. "Users often report newsletters as spam," says deGuerre. "We can show a new button in webmail, 'secure unsubscribe', and when the user clicks on it, Cloudmark will facilitate the unsubscribe and track which newsletter providers behave well and which sign you up for more spam."

"There's a new trend in newsletter spam," Champine warns. "Spammers highjack newsletters by taking a legitimate newsletter and embedding their own image or unsubscribe button so that it's impossible to tell the difference between that and the real newsletter."

"We can tell the difference by checking, for example, whether the newsletter is delivered from a familiar set of IP addresses," explains deGuerre. "Of course, we find a high occurrence of good unsubscribe practices."

He adds that the company works with LASHBACK, a site that has been tracking unsubscribe practices for several years.

The company has improved its policy based message management. "In the past," says deGuerre, "we had only two categories for bad mail: spam and virus. Now we have categories like phishing and newsletter and might add other categories in the future."

All of this is not a complete list of Cloudmark Authority 2.0's features, but it highlights much of the company's unusual and interesting technology.

Currently, however, the solution is targeted at deployment in large ISPs. "We're planning a solution targeted to small ISPs later this year," says Champine. "We feel that if we can prove ourselves in large deployments, smaller ISPs will have confidence in our solution."

Pricing and availability
Cloudmark Authority 2.0 is available now. Pricing was not disclosed at press time.

—End