Data duplication stifles quality

Mervyn Mooi

DATA QUALITY in an organisation will never be 100% accurate due to data duplication, missing content and huge volumes of unstructured data.

This emerged during ITWeb`s Data Warehousing 2010 event, in Midrand, where Mervyn Mooi, director of Knowledge Integration Dynamics, gave an overview of how companies can enforce data quality. "At the operational level, data quality should be at 100%, but reported at strategic level, the data quality may deviate slightly. Within this deviation, a business can allow for a deeper analysis into the data quality problems.

"Some banks allow for 5% deviation; however, it does vary from organisation to organisation and it depends on business priorities," he explained.

According to Mooi, organisations striving for improved data quality must create a data warehousing framework that needs to be controlled, governed and accessible.

"Typically, missing content will result in incomplete results," he said. "Differing content for the same entity will land up with many versions of the truth.

"A recent survey revealed that all organisations in SA have data quality issues in various aspects and have at least a 10% budget spent on fixing the data, [and] to make things worse, we duplicate data everywhere. Main entities are defined more than once with different formats and storage conventions."

Mooi added that data quality starts outside of the data warehouse and has to be an enterprise-wide task. "A business needs commonly defined data models and architectures need to be lean. If you address data governance, services, architecture and content, you should have little problems with your data warehouse."