Ron Dvir, ECI Telecom, 30, Hasivim St. Petah Tikva, Israel, eMail ron.dvir@ecitele.com
Dr. Stephen Evans, The
CIM Institute, Cranfield University, Cranfield, MK43OAL, UK,
eMail Evans @ cim.cran.ac.uk
There is a consistent gap between users expectations regarding Information Quality (IQ) and the perceived quality of the information they are using. An explicit approach to IQ is required, meaning that all stakeholders should specify in detail the IQ requirements, design them into the information solutions and track their fulfillment.
A Total Quality Management (TQM) based framework for the IQ improvement process is proposed. The framework employs six TQM concepts, namely Customer Focus, Leadership, Teamwork, Continuous Improvement, Measurement and Benchmarking. A case study about an initiative to improve information on Project Status is discussed.
IQ dimensions are at the centre of this framework. They are organized in a three level hierarchy. User satisfaction is decomposed into "Customer Needs" which are translated into "IQ metrics" These dimensions are treated as objects. The paper lists the set of operations that should be performed on these objects including selection, scaling and prioritization.
InfoQual, a methodology designed to facilitate the manipulation of IQ dimensions in the improvement process, is described. This methodology is based on the TQM framework and uses three specific tools:
In a survey recently conducted in the UK (Rolph and Bartram, 1994) managers and professionals from various disciplines were asked to evaluate the quality of information they were using. In the questionnaire, the notion of IQ was deconstructed into 8 dimensions. The responses are summarized in table A.
| |
| Accuracy | |
| Reliability | |
| Presentation | |
| Timeliness | |
| Completeness | |
| Information highlights main issues | |
| Relevancy | |
| Usable format |
The researchers have concluded that the perceived quality of information used by the respondents is not satisfactory. "It is just about adequate in some areas, but well below what is needed in world class corporations in other key areas"..
Another researcher states that "Many managers are unaware of the quality of data they use. Poor quality data appears to be the norm, rather then the exception" (Redman, 1995).
Disappointing results, considering that in the recent years there has been an increase in information-related investments, Information Technologies are rapidly advancing, and top management everywhere refer to information as a "Strategic Asset".
Perhaps objective measures such as PC processing speed or printer resolution have improved with time, and we are all connected to the Web. However, the gap between information users expectations and the perceived quality of information they use remains a problem. It has become evident that ever advancing Information Technology and larger capital investments may constitute necessary but not sufficient conditions to assure high Information Quality, as defined by the user.
Quality Engineering methods, such as TQM (Total Quality Management), SPC (Statistical Process Control) and QFD (Quality Function Deployment) have become commonly used by many product design and manufacturing disciplines, and are rapidly entering the service disciplines. In the field of Information Quality, however, the use of these tools is rare.
Numerous researches are addressing the problem of IQ dimensions. It is widely agreed that the complete Information Quality notion must be decomposed into more concrete dimensions, such as relevancy, accuracy etc. (Fox, Levitin and Redman, 1994). It is assumed that these dimensions should be used in the process of designing, testing, selecting or evaluating information systems. However, very few papers take a practical approach to the implementation issues: How should these dimensions be used? What process is required ? Who should do it ? How can quality dimensions be translated into measurable metrics ? How should they be selected and prioritized ?
The purpose of this paper is to present how well established quality engineering tools and concepts could be used to create a methodology that addresses these implementation questions.
In New Product Development projects, it was shown that investing more time on defining product specification and customer expectations would lead to significant reduction in overall Time To Market and a reduction in late and expensive engineering changes (Hauser and Clausing , 1988). It is proposed to apply the same principle to the case of Information Systems. An explicit (and effort consuming) investigation of user IQ needs will reduce overall Time To Market of the improved information solution, and, needless to say, reduce IQ expectations gap. This approach can replace the existing implicit approaches to IQ. In such implicit approaches, designers of information solutions appear to assume that conducting in depth functional system analysis (with tools such as Data Flow Diagrams) and using the latest technology assures that all IQ issues are somehow covered.
Total Quality Management (TQM) is a management approach aimed at satisfying all customer requirements, needs and expectations using a Continuous Improvement approach .The TQM principles can be grouped into the following practical and common sense concepts (Hari, 1995):
The framework is outlined in Figure 1. Its components are described in the following section.
This section describes briefly the role of each concept in the proposed IQ improvement process. A case study about Project Status Information accompanies this section in order to demonstrate the relevance of each TQM concept to IQ. Project Status Information should provide a clear and updated picture of the status of all project activities. The information therein is critical to all of the project stakeholders. In the case study, all opportunities to employ the TQM concepts to IQ were missed. It is argued that the lack of disciplined IQ approach has resulted in unsatisfactory results, namely no improvement in the quality of the Project Status information.
Concept: management should demonstrate leadership by:
(a) recognizing IQ as a strategic issue,
(b) allocating the appropriate resources to IQ improvement- capital, management attention, vision and priorities.
(c) Setting an example as
the first to require, use or provide better quality information.
This role is the responsibility of all management levels, from
the company president down to team leaders.
Case Study observations (concept misuse): Top management has acknowledged that Project Status Information is of poor quality. It has started an initiative to improve it. However, management support was limited, and after initiation it never invested the required management attention to the issue. Management did not really use the improved information in pilot projects and did not provide any feedback. When conflicts between investment in the IQ improvement and daily R&D work were uncovered, management did not interfere to resolve.
Concept: The modern quality paradigms emphasize the importance of customer satisfaction as a driver to the improvement process. IQ improvement efforts should focus on the identification of users, specification of their true IQ needs, and fulfillment of these requirements. The "voice of the customer" should lead the entire improvement process.
Case study observations (concept misuse): From the beginning, top management was considered to be the customer of the Project Status Information. Management perception on required information quality was examined and implanted into the solution. However, the analysis failed to recognize a second major customers' group, namely the individual project managers and R&D engineers. After the information solution was designed, it was found that the second group of customers had different and sometimes conflicting IQ needs. For example, visibility of the variance between planned and actual milestones' completion dates was a primary requirement of the management, but a clear disadvantage in the eyes of project managers.
Concept: Specification of IQ needs and metrics, as well as fulfillment control are based on teamwork operation. All stakeholders are included in the team. A typical team hosts representatives from the information users' group, information providers, information solutions' suppliers, information organization and other relevant parties. A certain level of management participation is required as well. All the above functions are responsible for higher quality of information.
Case study observations (concept misuse): The IQ improvement team included only members of the Quality Department. In a post mortem analysis it was recognized that this led to misunderstanding of customers' IQ needs, lack of cooperation with solution implementations, and disagreement on the quality of the new project information. At the post mortem, it was suggested that a team that represents the various functions would have achieved better results. The team should have included the following members: senior manager (as a customer), project manager (as a customer), R&D engineer (acting as both customer and information provider), budget and control manager, MIS department representative, a project management SW package supplier. The role of the Quality Management department representative should have been to facilitate the process.
Concept: IQ metrics are used to translate the information user needs into measurable specifications. These specifications should be designed into the information solution. Once the solution is provided, IQ metrics are used to asses the solution's actual performance against the requirements, and effectively against user needs.
Due to the special importance of this concept to the InfoQual methodology, a more detailed discussion is provided in section 3.
Case study observations (concept misuse): there was no effort to explicitly analyze IQ requirements and specify appropriate metrics. As a result the needs' analysis was not complete. Several quality aspects, such as accuracy and dependability, were ignored. When the pilot project was completed, there was no objective way to compare it with pre-defined metrics and targets, in order to asses the quality of the solution.
Concept: In order to achieve "world class" IQ, it is necessary to explore what IQ levels are achieved in the "external world". We refer here to other functions in your organization, other organizations in your industry or even other industries and professional domains. Benchmarking supports the IQ improvement team in setting high but realistic targets that energize the process. Benchmarking is also a useful tool to discover new and practicable metrics and methods to measure IQ.
Case study observations (concept misuse): Specification of IQ issues was done implicitly. It was vaguely stated that the Project Status Information should be "updated", "accurate", "visible" and "complete". However, there was no serious attempt to check whether these targets were achieved in other R&D organizations working in similar technological environments. At the end of the improvement process the above IQ targets were not met. It is argued that demonstration of high quality information on project status achieved in other high technology organizations could have been a motivating factor in overcoming the implementation difficulties.
Concept: In the field of IQ, quality improvement efforts are not a one time effort. There are two aspects to this concept: cultural and methodological.
(a) The cultural aspect: In a culture that promotes IQ continuous improvement, each member deals with the following questions: What is the meaning of high quality information? How is it defined and measured? Do I require, obtain and use high quality information? Do I provide such information? What must I do in order to get or provide better information?
(b) The methodology aspect: The cultural aspects of IQ are beyond this paper's scope. However, it should be noted that IQ culture cannot be achieved by having the company president stating "Information is critical, lets improve it continuously." Rather, it should be deployed via a series of practicable improvement activities. Implementing a methodology such as InfoQual can help create the common language and behavioral habits of an IQ culture. The InfoQual methodology is based on the PDCA (Plan Do Check Act) cycle, a popular model to organize the improvement process (Hari, 1995). The cycle is based on 4 phases:
Once completed, the cycle
is reiterated in order to achieve further improvements.
Case study observations:
Though the company invested a lot of capital in IT, and most of its operations were information intensive, the IQ culture was not a part of the company culture. Thus management could initiate an effort to improve Project Status Information and then abandon it. PLAN and DO phases were performed, but CHECK and ACT phases were neglected thus the improvement cycle was never completed. Users and providers of this information still do accept a very low quality information without actively trying to improve it. The PDCA cycle was not repeated as expected in a true continuous improvement mode of operation. Therefore the desired IQ improvement was never achieved.
The case study demonstrates that when properly applied, the selected TQM concepts are highly relevant to IQ. However, partial or wrong application, as was the case with the Project Status Information, can result in an IQ improvement process with good intentions, high resource investments but only little (or no) actual improvement.
The InfoQual methodology focuses on the Plan and Check phases of the PDCA cycle. Therefore IQ dimensions and metrics, on which these phases are based, are at the centre of the method. They are used to explore customer needs, to specify the desired information properties, to communicate the benchmarks and to compare the performance of the resulted product (i.e. information quality) against targets.
There is a growing body of research literature concerning the measurement of information quality. The Dutch company Cap Gemini Pandata (Delen & Rijsenbrij,1992). decomposes the entire information quality notion into four dimensions, 21 aspects and 40 attributes. They include this structure in the company procedure covering SW packages auditing.
Zmud suggests a set of 4 dimensions (information quality, relevancy, format quality, meaning quality) divided into 25 factors (Zmud, 1978).
In his research on value adding processes in information systems, Taylor lists six "user criteria" (ease of use, noise reduction, quality, adaptability, time saving, cost saving). Each criterion is then translated into several system attributes (Taylor, 1986).
AT&T is conducting comprehensive research on Data Quality, which is closely related to IQ. They identify four categories, namely accuracy, currentness, completeness and consistency (Fox, Levitin and Redman, 1994).
The TDQM Institute (Wang, Storey and Firth, 1995) has evaluated the research literature on dimensions of data quality. They conclude that there is a lack of consensus on what constitutes a "good" data quality dimensions set. Furthermore, there is no agreement on the definition of seemingly simple dimensions such as "Accuracy." Three research avenues are suggested:
(1) to use a scientifically grounded approach in order to rigorously define information quality dimensions.
(2) to create a universal standard set of operational quality dimensions.
(3) to let information quality dimensions be defined by the information customer(user).
The approach taken by this paper is similar to the third avenue. It suggests that the definition of IQ dimensions and their measurement should be done as a teamwork effort by information users, information providers, information solution suppliers and other relevant stakeholders.
There is a general agreement in the literature on the need to decompose the Information Quality notion into an organized hierarchy. Numerous classification and hierarchical methods have been described. A triple level hierarchical organization is proposed here (figure 2).
Figure 2: Hierarchical
organization of IQ dimensions.
The InfoQual methodology is designed to support the team in defining, prioritizing and performing other operations on these two object types, namely IQ Users Needs and IQ Metrics. These operations are listed in Table B.
| Operation | Description | Output Examples |
| EXTRACT needs | Find out what are the users' IQ needs, expressed in his words (voice of customer) | Timeliness, Relevancy, Accuracy |
| TRANSLATE needs | Translate abstract needs into concrete metrics. | Need= Accuracy
Metrics= error rate, precision |
| DEFINE metrics | Define how the metric will be measured (definition, unit, scale, data source) | Error rate ,defined as % of documents with at least one erroneous field |
| ANALYZE metrics | Explore interdependencies between metrics (tradeoffs, synergy) | Tradeoff between Completeness & Response time |
| PRIORITIZE metrics | Rank metrics by importance, according to user needs and other considerations | In application X importance order is: Response time, error rate, format standardization |
| MEASURE current performance | Measure the current IQ performance | Response time= 8 days (average) |
| BENCHMARK performance | Explore IQ performance in other organizations. | In company Y: Response time= 4 hours. |
| SET TARGETS values | Determine target values | Response Time= 3 hours |
|
NORMALIZE metrics | Each metric has different scale and unit. This operation creates a common scale, in order to enable the evaluation of the overall IQ performance (integrated IQ index) | Performance Scale:
5-outstanding 4-good, 3-acceptable 2-poor |
| COMMUNICATE performance and targets | Communication of metrics information to all stakeholders, including management | Graphical report |
This section sets the objective of the InfoQual methodology, identifies its users and presents its tools.
The objective is to facilitate the PLAN and CHECK phases of an IQ improvement project. The goal is to enable superior completion of the customers IQ needs and expectations.
InfoQual is an organized improvement process in which all customer needs are identified and translated into IQ metrics that are tracked throughout the project.
The scope of the methodology is limited to the IQ "WHAT" issues: what IQ needs are to be treated, and what are the required performance levels. It does not specify HOW these improvements can be achieved.
In order to design a practicable methodology, it is essential to define its potential users. Then, the special needs and concerns of each group must be identified.
Appropriate implementation of the methodology involves a teamwork operation, with representatives from several groups. Each can initiate the process, facilitate and lead it, or play an active team role. In the following section, the potential users are defined:
The process used in InfoQual is based on the TQM framework to IQ improvement, described in section 2 (see flowchart in Figure 1). IQ needs and metrics are at the centre of the framework. Three tools to manipulate these objects are integrated into the TQM framework in order to form the methodology. These tools are QFD, Metrics database and IQ graphical presentation. Table C maps these tools onto InfoQual operations. Then, the following sections briefly describe these tools.
| Operation | Tool | Notes |
| EXTRACT needs | QFD | This is the first stage of a QFD workshop. |
| TRANSLATE needs | QFD, METRICS DATABASE | The database is used as reference to available and pre-defined metrics in order to facilitate selection of IQ metrics reflecting the customer needs |
| DEFINE metrics | METRICS DATABASE | The database facilitates the precise definition of the selected metrics (units, scales, etc.) |
| ANALYZE metrics | QFD | This is where the QFD methodology is used to analyze the correlation between needs and metrics (positive strong, positive medium, positive weak, negative strong, negative medium, negative weak) |
| PRIORITIZE metrics | QFD | The QFD process prioritizes the metrics according to their overall contribution to customer needs satisfaction. |
| MEASURE current performance | Standard measurement tools | Measurement tools should be selected according to the metric. The metric definition in the database specifies the appropriate measurement tools (i.e., customer satisfaction surveys, SPC charts, etc.). |
| BENCHMARK performance | METRICS DATABASE | Benchmarking data on some metrics is included in the database. |
| SET TARGETS (target values) | QFD | The stakeholders use the QFD data (benchmarks, priorities etc.) in order to rationally set target values for each metric. |
| NORMALIZE metrics | METRICS DATABASE | The database specifies normalization scales and formulas for each metric |
| COMMUNICATE performance and targets | IQ GRAPHICAL PRESENTATION | Used throughout the improvement project cycle life |
QFD (Quality Function Deployment) is a structured process to design products or services based on the customer needs. A detailed description of the method is beyond the scope of this paper. It is given, for example, in a classic article that has uncovered the beauty and rationale behind this Japanese method (Hauser and Clausing, 1988). During the process, the design team members walk through the elements of a matrix called "The House of Quality". This method executes many of the operations on performance metrics discussed in section 3. In particular, it is strong in translating the customer needs into concrete product attributes. The method facilitates the identification of tradeoffs between attributes and also allows their prioritization. Therefore QFD was selected to perform IQ needs and metrics' manipulation.
In QFD, it is common to use brainstorming techniques in order to identify attributes (IQ metrics in our case). This exercise becomes very difficult when dealing with abstract needs related to the information field. How would you measure "Information shareability" or "Relevance" for example? The challenge is to specify a metric that is relevant, significant and measurable. In many QFD cases, unsatisfactory metrics are used, leading to sub-optimal results. Thus, it makes sense to create a mechanism that documents IQ metrics once the improvement project is completed. In future improvement projects, these metrics can be extracted and reused.
The proposed company-wide database holds a record for each successful metric that was specified and used in IQ improvement projects. Each record includes complete information on the metric in order to support its implementation. Indexing mechanism supports the searching operation.
The metrics database does not replace the necessary team discussion devoted to metrics' identification during the QFD process. Rather, it supports this discussion by suggesting candidate metrics, shortening the time invested on this phase and, most importantly, improves the quality of the selected metrics. Similar idea is presented in a research listing about 180 candidate data quality attributes. These are used to stimulate thinking by a design team that specify data quality requirements (Wang, Kon and Madnick, 1993).
It is further suggested that the same approach can apply to an inter-company IQ metrics' database. The concept is outlined in Figure 3
Information on IQ performance should be effectively communicated within the improvement team, as well as to external functions such as management. It is necessary to report four values for each IQ metric:
Furthermore, it is desired to visualize an index of the integrated quality of information. A graphical presentation, based on spider charts, is selected. This presentation can be produced automatically from the values stored in the QFD's matrix.
This paper has proposed a TQM based framework to IQ improvement projects. Six TQM concepts are integrated into an organized process that translates IQ customer needs into IQ metrics and ensures their fulfillment. Metrics and measurement of information quality are at the heart of the process. These metrics are manipulated through a set of operations like selection, prioritization and presentation. Three tools that facilitate these operations are integrated into the TQM framework, creating the InfoQual methodology.
The methodology represents an explicit and manageable approach to the improvement of information quality. It suggests that the stakeholders should invest meaningful effort in explicitly defining and tracking IQ needs and metrics, using a formal and structured process. More research is required in order to test its usefulness. The following questions arise: what is the cost of implementation? How can effort be minimized, in order to make it acceptable to all stakeholders? What are the direct effects on information quality? What are the side effects and benefits? In what information domains is the methodology more practical?
Many quality approaches are
added on top of the regular business processes without eliminating
or replacing any existing activity. The pre-quality way hangs
around with some extra work in the form of the new way (Greene,
1993). Such redundancy must be avoided. Therefore, the InfoQual
methodology is designed to be integrated into common and
existing operational approaches to information systems specifications.
The next phase of this research will test the practicality of
InfoQual in real-life implementations.
References
Delen, G. P. A. J. and Rijsenbrij, D. B. B., "A specification, engineering and measurement of information systems quality", Journal of systems Software, 1992, Vol. 17, No. 3, pp. 205-217
Drucker, P. E., "The information executives truly need", Harvard Business Review, 1995, No. 1, pp. 54-62
Fox, C. ,Levitin, A. and Redman, T., "The notion of data and its quality dimensions", Information processing & management, 1994, Vol. 30, No. 1, pp. 9-19
Greene, R. T., "Global quality- A Synthesis of the World Best Management Methods, 1993, New York, ASQC Quality Press, pp. 117-118
Hari, A. (editor), "The Quality Terms Lexicon", 1995, Israel, Quality and Excellence Center, Prime minister Office (Hebrew text).
Hauser, J. R. and Clausing, D., "The House Of Quality", Harvard Business Review, May-June 1988, pp. 63-74
Redman, T.C., "Improve Data Quality for Competitive Advantage" ,Sloan Management Review, Winter 1995, Vol. 36, No 2, pp. 99-107
Rolph, P. and Bartram, P., "The information agenda: Harnessing relevant information in a changing business environment", 1994, London, Management Books 2000, pp. 65-87
Sargent, P., "Data quality in material information systems", Report CUED/C-EDC/TR2, Engineering design center, 1991, Cambridge university
Taylor, R., "Value added processes in information systems", 1986, New York, Albex Publishing, pp. 48-70
Wang, R., Kon, H. B., and Madnick, S. E., "Data quality requirements analysis and modeling", 9th International Conference on Data engineering 1993, pp. 670-677
Wang, R., Storey, V. C. and Firth, C. P., "A framework for analysis of data quality research", IEEE transactions on knowledge and data engineering, 1995, Vol. 7, No. 4, pp. 623-638
Zmud, R., "Concepts, theories and techniques: an empirical investigation of the dimensionability of the concept of information", Decision Design, 1978, Vol. 9, No. 2, pp. 187-195