Dealing with dirty data

More than ever, data quality is a major issue for South African financial services companies. Success is not dependent on collecting the most data, but on collecting data that is accurate and error-free. To ensure this, organisations need to measure the quality of their data effectively.

"Raw data in itself is meaningless," says Kerry Evans, General Manager: Financial Services, of SAS Institute, leaders in business intelligence. "However, it forms the vital foundation for obtaining real business intelligence on which strategic decisions can be based. Dirty raw data obviously leads to distorted information."

Last year, an independent survey of business users in Europe, commissioned by SAS, found that two-thirds (66%) of respondents said that company profitability was affected by the accuracy and completeness of data, highlighting the critical business implications of 'dirty data'.

The survey also found poor data quality is a widespread phenomenon. 53% of respondents felt that between 1% and 10% of their data was inaccurate, with 10% of respondents admitting that up to 33% of their data was flawed.

According to a recent Gartner research note (December 2003), financial services providers that measure the quality of their data will be three times more likely than their counterparts to report high data quality by 2006 (0.8 probability).

"This should encourage local financial services companies, most of whom are currently tackling data quality improvement projects as a result of increasing fraud threats, and new regulatory requirements," says Evans.

Gartner points out that benefits of measuring data include identifying the suitability of data for its intended uses. Data measurement results are also critical for justifying and evaluating data improvement initiatives.

Trek to the summit

00Days

:

00Hrs

:

00Min

:

00Sec

"As data volumes grow - one major SA bank, for example, has a data warehouse that is 5TB - so manual reviews of data quality become useless," she says. "Some companies outsource data quality measurement, but this effectively removes data control from the hands of the company, and does pose security issues."

According to the Gartner research note, organisations that do have effective data measuring processes in place tend to measure only those errors that are simple to spot, such as missing fields and duplicate records. More subtle errors, such as name misspelling - which can alienate customers - or incorrect credit score that results in a bad loan decision, can have a direct impact on the fitness of data for key business processes, such as customer relationship management, risk management and financial and regulatory reporting.

"Sophisticated business intelligence tools help organisations look way beyond the obvious errors," says Evans. "What is important is to identify all errors - from the obvious to the most subtle, something that can only be achieved by a tried and tested solution, that includes some kind of feedback loop and is part of a strategic plan within the company. It is important that once subtle errors are detected, processes are put into place to ensure that these cannot happen again."

SA's major banks are all focusing on data quality. As Vicky Ratsoana, head of knowledge discovery and data mining at First National Bank (FNB) says, information quality is the critical element in any banking environment. It needs to be addressed by all stakeholders in the enterprise, and positioned as the most valuable strategic resource, second only to human capital.

He says that when it comes to data quality, "we are all stakeholders and it's in our own interests to have good quality data. Information must be top of mind for everybody concerned."