Testing in the anti-malware industry has heated up since the formation of the Anti-Malware Testing Standards Organisation earlier this year. This is according to a Kaspersky Lab report: 'Repercussions of dynamic testing'.
Roel Schouwenberg, the report's author, says 'dynamic testing' is the introduction of malware samples into a test system, with the intent to execute them. It is a time-consuming process even when automated, and the need to avoid virtual machines for the sake of valid test results also contributes to the time requirements.
The focus of dynamic tests on detection rates is outdated, he warns, and testers now need to consider a product's protective measures as well. Because the sample set used in dynamic testing is relatively small due to the amount of time required to investigate each one, each sample needs to be highly relevant, he adds.
Schouwenberg argues that results published on the efficacy of these important samples provide malware authors with valuable information on how their creations fare against existing security products. This can then be used to determine the areas malware creators need to focus on in order for their creations to defeat the detection algorithms in use.
While big-name AV testing site Virus Bulletin does not yet publish dynamic testing results, he claims it plans to use information from its upcoming prevalence table to pick samples for testing. He is concerned that while testers will likely start out by only mentioning malware families, they may end up disclosing specific malware names.
AV-Comparatives, another big-name testing site, does not publish dynamic testing results, but Schouwenberg's report states it intends to publish the names of samples being used in future dynamic tests.
He believes this approach should be avoided, and testing sites and companies should rather take a leaf out of AV-Test's book and prohibit magazines from disclosing the names or hashes of malware samples used in their testing, but still share the data with AV vendors that participated in the test.
While this approach may not be particularly transparent to the end-user, the report says, the risk mitigation it offers is far preferable.
Dynamic testing and the publishing of its results can have very real consequences for malware authors and may indeed shape their future actions, Schouwenberg says. Security vendors need to keep this in mind in their quest for education, lest they lose sight of what is really important: the protection of users.
Share
