Bots can skew your analytics: Is your data lying to you?

Anton Jacobsz, CEO at Networks Unlimited Africa.

---

Data and analytics are increasingly important tools that help businesses decide where to invest and develop. Take UPS, for example: it has used data/analytics to save up to $400 million every year through optimising delivery routes.

"While we rely on big data to provide actionable business intelligence on everything from delivery logistics to crop yields to innovative marketing techniques, the information we gather is only as valuable as the context it is measured against," says Simon McCullough, major channel account manager at F5 in SA. "We all know that data can't really lie, but it can paint a distorted view depending on how it is interpreted. It's the old 'garbage in, garbage out' maxim and, in this scenario, the 'garbage in' aspect of the equation can be boiled down to one main culprit: bots."

A recent F5 white paper says a large percentage of Web traffic to a business's site is probably from bots and, while bot traffic, whether malicious or benign, may appear to be legitimate, users should differentiate scans and probes performed with automated programs from real engagement from customers.

Anton Jacobsz, CEO at Networks Unlimited Africa, a value-added distributor of F5 in Africa, says clicks from bots are not the same as clicks from actual humans.

"These unwanted clicks can skew analytics, distort market intelligence and hurt the bottom line," he says.

"It sounds a bit dramatic, but consider company X, whose customer base is located in SA and whose data keeps telling them there is an influx and a continuous rise in traffic from outside the country. Company X might be inclined to invest resources into expansion to a much larger international market when, in reality, most of that traffic is just scanning and scraping its pages.

"The bottom line is that analytics cannot be accurate if companies are not proactively and effectively managing bots and their interactions with sites and services."

Bots cost businesses money

F5's McCullough agrees but adds that bots can do more than just deliver bad data that affects businesses decisions. "They can negatively affect your bottom line," he says.

"With the growing popularity of cloud-based services provided through a utility billing model, automated traffic can drive costs up without providing any business value."

The paper urges readers to consider the following: "Every bot request against your site can be quantified in terms of bandwidth, CPU time and memory costs, all of which will show up on your monthly bill. More bot traffic means more compute power in the cloud, more machine instances, and higher costs.

"In addition, there are a variety of malicious bots out there. Criminals use automated programs to steal pricing to boost competitive intelligence, disrupt business through denial of service, and attempt fraudulent transactions using a variety of means, all of which can negatively affect the reputation of your business and your brand. Not to mention that you're actually paying to render pages and perform server requests for machines that are actively trying to hurt your business. These aren't potential customers; they're attempts to scan for vulnerabilities or steal intellectual property."

Says Jacobsz: "Whether it's in skewing data sets, increasing utility costs or acting as vectors for malware, bots can cost businesses money. Some business may opt to ramp up security measures but if these are ramped up too high, these same businesses risk exceeding customer tolerance, effectively DOSing (denial of service) their own Web sites and, since blocking all automated traffic isn't an option, businesses must investigate how to best facilitate good bots while mitigating damage from the bad ones."

Smart WAF; boosting the bottom line

Smart Web application firewalls (WAFs) can manage the traffic to Web sites, delineating and filtering automated visits from engagements with actual humans. Eliminating the large swatch of bot traffic results in better data, which in turn delivers better business intelligence.

An effective WAF with bot-management capabilities can optimise Web properties by reducing useless traffic, resulting in considerable cost savings. Users can streamline by serving only real and potential customers, which means that their security tools are providing real value in decreasing costs in the cloud.

"Smart security solutions can help businesses manage automated traffic while cutting costs and improving overall security," says Jacobsz. "With a combination of bot management and application protection solutions that learn and adapt to the shifting threat landscape, users can better serve human customers while optimising their sites and services, all while getting the data-driven insights their businesses need."

To find out more, please contact Esti Bosch, F5 product manager at Networks Unlimited: esti.bosch@nu.co.za.

You can access the F5 Networks report: "Get the Clicks that Count", here.