Spam-filter testing flawed, according to META Group

As organisations continue to battle the proliferation of spam in the enterprise, the importance of choosing the right e-mail hygiene vendor becomes increasingly apparent. In recognising the critical need for spam-blocking tools, organisations and publications have rushed to develop testing processes in order to segment the vendor landscape and evaluate product offerings. According to META Group, a leading provider of information technology (IT) research, advisory services, and strategic consulting, while the importance of testing cannot be overlooked, the type of testing in which many organisations have engaged has been flawed, at best.

"At root, the issue with many of the most common testing techniques is that they do not simulate real-world conditions or replicate real-time anti-spam abilities," said Matt Cain, senior vice president with META Group`s Content & Collaboration Strategies service. "Simply put, current testing techniques do not allow organisations to accurately evaluate product capabilities. Invalid testing can, and will, lead to inappropriate vendor selection, an unhappy CIO, and more spam than anticipated."

META Group analysts warn, in particular, about testing performed on forwarded mail; specifically, they discredit evaluations that forward a large quantity of mail to a specific destination for filtering purposes. Such testing methods ignore the sophistication of header information and IP address manipulation, among other aspects.

Les Stevens, practice leader: Security and Risk Strategies for META Group, highlights, "In a recent assessment META Group identified more than 60 vendors vying for a slice of the e-mail hygiene pie and each claiming competitive advantage over the others. As SPAM continues to be a problem in South Africa, organisations will need to be specific in defining their e-mail hygiene requirements, including configuration and operational requirements, so that when comparative testing between vendors and solutions is done it is on a measurable and equitable basis.

These methods also do not allow for real-time message checks, including sender Internet Protocol (IP) validation tests. Finally, using this approach, anti-spam filters cannot glean intelligence from the SMTP transaction, a critical data point for many detection techniques, including traffic-shaping technologies, sender reputation services, and e-mail authentication protocols.

"Spam-filter evaluations based on a mail-forwarding scenario will not provide testers with the information they seek about the accuracy of an anti-spam tool in the workplace," continued Cain. "Equally as ineffective as the forwarding approach are evaluations that focus exclusively on spam capture rates and false-positive generation. Testers must learn to look beyond the easy statistics and measure end-user satisfaction, ease of use, and operational control."

A best-practice approach to evaluating e-mail hygiene solutions, however, goes beyond these broader metrics. Smart testers, say META Group analysts, will segment the vendor landscape into the three major delivery models (hosted, appliances, and traditional software load) and compare accordingly. Testers must also recognise that, like many technologies, spam-filter products will not provide optimal performance in a plug-and-play situation. Instead, testers need to do the appropriate tuning, enabling recipients to set up block/allow lists to attain a more accurate picture of blocking effectiveness.