Bloggers shred blog survey

Everywhere you turn, people bombard you with statistics. They're in every sales presentation, at every conference, and in every second news article.

But, survey after survey shows that surveys are junk.

Parse that sentence, if you will. A statement such as this has several meanings, at least one of which could be correct. The problem is, which is intended, and which is correct?

A recent survey of South African bloggers offers a number of examples of typical problems with surveys. Some were spotted by Eve Dmochowska, a sharp-eyed Web strategist. Others were noted by survey respondents such as bloggers Tertia, Tracy Stokes, and Chris M, each of whom claimed to have answered that they earn more than R3 000 a month from blogging, while the survey says none of the respondents did.

In fairness, the survey's creator, Alistair Fairweather, rapidly responded, copped to the problems, and probably saved the survey's future credibility with his response.

Still, some of the results, or their interpretations in the presentation document, raise more questions than they answer.

Take the first page. A highlight bubble says 58% of respondents are between the ages of 25 and 44. This appears to be true. It's also true that 52% are 35 or over. Why is the former a highlight, but the latter, which suggests more than half of all bloggers are fogeys who have less than half of their allotted seven-score-and-ten years left, is not?

Number of children is just confusing. Apparently, 46% of bloggers have them (all parents should be this cool). Conversely, 54% of respondents don't. But 80% have three. This looks like a simple error in the data, but such errors don't add to the credibility of a survey.

Another highlight box says "many do not receive any comments". "Many" being 9% of the respondents. Someone else might have said that almost all blogs do receive comments.

Apparently, "the vast majority of bloggers only have time for a contribution to one blog". Without nitpicking that having one blog and having time for one blog are two different things (or that "majority" is a singular noun), here's the real problem: 48% isn't a "vast majority". It's "less than half". In fact, a small majority (52%) has more than one blog.

Some answer options are very subjective, or introduce bias. Would you call yourself a "deep thinker" (also known as a "pretentious git"), or do you prefer "voice of reason"?

Ivo Vegter, Freelance journalist and columnist

The options for categorising blogs are "personal (topics relevant to me and my life)", "professional (about my industry and profession)", or "corporate (on behalf of my company or organisation)". Where would you put political opinion? Or sport commentary? Or environmental activism? Are they just personal?

Bloggers are "not, in the main, getting other perks". In fact, 24.2% of bloggers say they get free products, services or invitations. A quarter of all bloggers is a lot of people who get free stuff for blogging.

Some answer options are very subjective, or introduce bias. Would you call yourself a "deep thinker" (also known as a "pretentious git"), or do you prefer "voice of reason"? A plurality of respondents seems to prefer the faux-modesty of "man or woman on the street". There's a world of value judgements that affects the response to such a question. And what's the difference between "rational" and "intellectual", in terms of blog style? Together, they trump all other styles, but separately, they lose out to "humorous". Is "intellectual" or "rational" mutually exclusive with "humorous" or "controversial" or "sarcastic"? Why? Call me obtuse, but I can't draw any conclusions from such vague descriptions.

The commentary contained in highlight bubbles is important in a survey, because inevitably, this is what gets quoted in media reports and sales presentations. That it accurately reflects survey results is no less important than that the survey accurately reflects the actions and opinions of the sample, or that the sample is representative of the target population.

To show how easy it is to manipulate survey results, consider the following (fictional) question: "Do you care about the environment?" As options, offer "yes, a lot", "yes, a little" and "not at all". Let's say the result comes back with a 30-60-10 split between the answers. If you're an environmental group, you could claim that 90% of respondents care about the environment. If you're opposed to environmental regulation, you can claim 70% of respondents don't care very much. You could further bias the results by simply rephrasing the options. Replace "yes, a little" with "not very much, no", and you're likely to get a different split, which you can still spin any which way you like. For more advanced trickery, introduce bias by adding comparators: "yes, but less than I care about my family".

Here's a real-life example, spotted by journalist Phillip de Wet. A certain company published a survey, asking: "How do you view the cost of [the problem we solve] today versus two years ago?" The options are 'more than double', 'double', 'marginally higher', 'the same'.

This neatly excludes from consideration the (very likely) possibility that the cost of what they're proposing to sell you a solution for is, in fact, lower today. Whatever the results, the survey is designed to have only one outcome: to justify investing in their product.

Why don't survey designers use Likert scales for questions such as these? Is it because they don't know what a Likert scale is? Or because a Likert scale won't give the required results?

Perhaps I'm picking on Fairweather unfairly. I'm not suggesting he intended to manipulate the results of the SA blog survey. On the contrary, it is a very good idea, he clearly put a lot of effort into it, much of it is useful and enlightening, and it has the potential to become even more valuable over time.

However, the point stands: designing and conducting a good survey is a lot harder than it looks, as many of the respondents to the survey are pointing out, scathingly, on their blogs. Good luck with the new, improved version.