"Data Quality"- What , Who and how?

A data set which meet these below 4 conditions is a quality data set.

(A)Complete: must have all attributes, measures ( as agreed with customers).

(B)Correct: each attribute must carry right values.

(C)Consistent : each attribute of the data set must carry same meaning if it appears elsewhere in the enterprise either in the same form or any derived form.

(D)Available on time (as agreed with customers).

In other words, compromise on any one or more conditions produces poor quality data. Poor quality data does increase development and/or maintenance costs of adding new data, modifying or maintaining existing data.

What explains the existence of poor quality data? Either or combination of following reasons

1.Misaligned goals between Top Management and middle management.(Lack of vision, ability to translate that vision to execution and actions by Top Management team)

2.Lack of technical skills

Stakeholders responsible for data within an enterprise broadly fall under 2 categories:

(1) Middle Management and team members- Responsible for meeting conditions A, B part of C and part of D
(2) Top Management and Sr Management: Responsible for meeting largely conditions C and D

Misaligned goals between these 2 categories of people leads to sub optimal decisions. It is not uncommon that -sponsors and managers of programs- choose to compromise on quality as they do not think incremental costs of achieving required consistency and latency are justified. It's the top management's responsibility to understand importance of data quality, openly acknowledge and periodically communicate its importance to entire organization, align goals , and to provide right organization design with focus on data governance, data management initiatives, and best practices.

Before you implement a BI solution…

Below pointers would be help you if you are considering a BI solution.

1. DIKI- Data-->Information--> Knowledge --> Intelligence.
Where you are on this progression line, where do you want go from where you are? Determining where you are could be tough. Stable operational/transaction processing system(s) would meet your data and most information needs. A rightly implemented DWBI solution significantly lower your costs to address complex informational needs and also serves you as a platform to deliver reliable, faster and low cost Level 3- analyses, prototypes, and to let you build Level 4- forecasting and predictive models.

2.Design Organization: Consider following functions - Operations/Infrastructure: Hardware, System software, Database, Feed monitoring teams. Data Architecture: Data modeling & standards, Engineering: Development teams, Standards, Monitoring and support teams, Data governance, stewardship-Business & IT, Data Quality/Arbitrage.
Lack of a particular function/standard poses challenges at a later point in time as your DWBI evolves. At times you can combine some functions based on size of your organization.

3.How will you reach where you want to reach?

(a) Which service provider/Partner will you choose if you’d like to outsource? : Evaluate partners. Rely on your network. Evaluate cultural fit. Rest will fall in place.

(b) Which tools you need? Evaluation of any ETL or BI tool takes 3-4 weeks. Take your partner's advice. Don't rush and buy every fancy thing out there in the market. Buy only what you need; don't go for exotic tools and appliances if you don’t need to. Remember tools cannot bring about Business Improvements; those tools are as good as you want them to be.

4.Define success: Medium term and long term for DWBI solution. Define quantitative metrics and intangibles. Cost benefits analyses, ROI over a period of 2 quarters to 20 quarters. This is very important. Usually no 2 stakeholders give same responses when asked about the success of recently deployed (<8 quarters) DWBI.

DQ and Data Information Knowledge and intelligence( DIKI)

Jan 12 2008: There has been recent surge of interest in Data Integration- CRM, CDI, MDM. So far the Data qaulity management - identifying and resoliving DQ issues has been largely limited to set of applications. A successful data integration should always be supported by and built on a strong DQ management established at the enterprise level; in other words, there will be and should be an increasing focus on data goverancne and data stewardship.