Beware what you write - your lie could be showing

Polygraph - lie detector - tests are widely-used tools of the trade for catching crooks. But in today's virtual world, many crimes take place remotely.

And in SA, the constitutionality of polygraph testing, particularly as a means to prevent or detect crime, remains questionable.

So imagine the potential of a technology that could determine whether or not an individual is lying - simply by analysing the language used in that person's e-mail message, affidavit, letter of complaint, insurance claim or job application.

"This is not as far-fetched as it sounds," says Bill Hoggarth, MD of the local subsidiary of the global SAS Institute, a leading provider of business intelligence which recently announced plans to introduce an advanced text mining product, SAS Text Miner, in mid-2002.

Text mining should not be confused with data mining. Until now, data mining has been largely focused on structured, record-oriented data, but the bulk of data in real-world organisations is unstructured text.

Text mining is the process of automatically extracting and analysing information from large collections of unstructured text documents such as e-mail messages, customer survey responses, sales call reports and Web site content. This data usually goes unanalysed - and very often unread - due to its sheer volume.

According to Hoggarth, companies implementing analytical customer relationship management (CRM) programs could use Text Miner to manage large volumes of inbound customer e-mails by categorising them for quicker action and follow-up. The software will be able to identify whether responses contain "angry" text or complaints by comparing responses to historical information.

In addition, Text Miner could sift through thousands of CVs and applications to properly match applicants' qualifications with company job vacancies.

But can Text Miner detect whether or not an applicant is lying on his or her CV, or bank loan application or insurance claim?

"In theory, yes," Hoggarth says. "Of course, the software itself can't tell if someone is lying. What it can do is look for use of words, patterns of words or choice of words that have been determined by a human - a behavioural psychologist, criminologist or even human resources professional - to indicate a likelihood of lying.

"SAS has not yet worked with any potential customers to test the viability of Text Miner software as a lie detection tool.

"However, SAS believes that Text Miner could potentially be used in the lie detection arena by leveraging its ability to identify patterns and associations in text which humans have determined indicate dishonesty on the part of the writer.

"Were such an application to be developed, it would most likely start by applying the software to texts that are already known to have been transcripts from conversations with people who have been identified as lying. The software would look in these texts for patterns and associations in the way language is used. A human would then look over those results and decide which could actually be taken as an indication of lying," he explains.

Text Miner, however, was initially designed for general purpose, commercial applications where organisations need to analyse large volumes of unstructured, text-based data in order to derive intelligence from them.

"Traditionally, a person would have to sift through text-based documents one-by-one - an impossible task with exceedingly large volumes of data. With text mining software, the filtering happens at a high rate of speed with fewer errors and with the ability to perform many different kinds of analysis.

"So, at this point in time, lie detector software is not available. But a company like Compaq is already using the solution to analyse more than 2.5GB of free-form text notes collected by e-mail and by the company's telesales representatives," Hoggarth concludes.