Acknowledgments Work reported herein has been supported, in part, by MIT's Total Data Quality Management (TDQM) Research Program, MIT's International Financial Services Research Center (IFSRC), Fujitsu Personal Systems, Inc. and Bull-HN. The authors wish to thank Stuart Madnick, Henry Kon, Jacob Lee, Reddy Prabhakara of MIT, Tom Redman of AT&T and Don Ballou of SUNY for their insight on earlier versions of this paper.

ABSTRACT: Information is recognized to be part of competitive advantage. Unfortunately, the quality of much data in information systems remains poor. Missing, inaccurate or inconsistent data can lead to business inefficiencies or missed opportunities. Data quality is an interdisciplinary problem; many concepts exist that can be applied to this issue. We have found that using an analogy to manufacturing, together with ISO 9000, an international standard for quality management, provides a comprehensive framework for the discussion of data quality. In this paper we survey research by authors who explicitly recognize a data quality problem. For each element of the ISO 9000 quality system we summarize its associated guidelines, survey related work from existing data quality research and point out areas unexplored or unresolved. The main contribution of this paper is to gather together the dispersed data quality literature and to act as a primer in data quality research.

1. THE STUDY OF DATA QUALITY

1.1 The data quality gap

Dramatic technological advances over the last two decades have increased our capabilities to acquire, store, process, and distribute data. Organizations are dependent on information stored in databases as a strategic necessity or to gain a competitive advantage. However, most databases are not error-free, and some contain a surprisingly large number of errors. Data manipulated by information systems may not only be "inaccurate" or "wrong" but may also be missing, out-of-date, inconsistent or otherwise inadequate for the specific purposes of the user.

There is much anecdotal (and often humorous) evidence for this. For example, the Hartford Courant [Courant, 1992 #711] reported that grand jury database records showed that the entire population of the town was dead. The town name had been entered in the wrong field, forcing the final character, 'd', into a field denoting prospects' status. There have been many reports in the literature suggesting that effective business decisions depend on good data quality and that poor data quality can be costly and sometimes disastrous [Johnson, 1981 #401; Laudon, 1986; Liepins, 1990 #509; Liepins, 1982 #537; Morey, 1982; O'Neill, 1988 #697; Zarkovich, 1966 #505]. Thus, the IT community needs to understand the causes and remedies of poor data quality. Just like product quality, data quality needs to be managed well if firms are to sustain good business performance.

Many organizations are aware of this relationship and may acknowledge that they have data quality problems; information systems professionals are able to voice their concerns about data quality but only in qualitative terms. Despite this recognition there is a relative lack of products and services in the market that deal with the data quality problem [Bulkeley, 1992 #737; Knight, 1992 #594]. Another limitation is the lack of theory, methods or conceptual frameworks with which to judge, measure, or improve data quality. Similarly, there is no treatment of the management, economic or organizational aspects of data quality.

1.2 What is data quality?

Before one can address issues involved in managing data quality, one must first understand what data quality means. Depending on the specific uses the same data may have different levels of quality. Data consumers may place varying importance on quality factors such as accuracy, completeness, interpretability and timeliness. Others may simply characterize data quality as the proportion of erroneous data. It may also be important to distinguish among quality factors of data values, of the data model and of the data representation. Another important consideration may be the boundary between quality of data and quality of the information system.

Thus, data quality is not easy to define or measure. Yet a quality measure or metric underpins any data quality standards, tools or assurance criteria. A standard definition of data quality remains an important unresolved problem in data quality research. The following sections discuss the meaning of data quality in more depth.

1.3 A framework to study data quality

There exist many concepts in the literature that can be applied to data quality and its management. For example, it is possible to consider data as products. This analogy allows the application of product quality management frameworks for data quality management. It also allows researchers to frame the problem in terms of prevention, repair and detection [Juran, 1989 #678; Deming, 1986 #374; Crosby, 1986 #478; Pall, 1987]. Others use the concept of a data life cycle, where the sequence of activities from creation to consumption of data is considered [Wang, 1993 #564]. Another concept is the value chain, where originating, distributing and consuming organizations are categorized by the value they add at each stage in the production process [Porter, 1985 #247]. Some think in terms of an EDP audit [Mollema, 1991]. Yet others like to regard the problem as one of database integrity [Date, 1981 #526; Codd, 1990 #717].

All of these views have merits. However, we choose to use the data manufacturing concept structured by an existing manufacturing quality management standard. We believe this combination provides the best form and rigor to our discussions of data quality. Moreover, we suppose that the fundamental analogies that exist between quality issues in a manufacturing environment and those in an information systems environment are sufficiently strong to support our conclusions.

Manufacturing can be viewed as a processing system that acts on input material to produce output material, or, ultimately, finished goods. In this way an information system can be viewed as a processing system acting on input data to produce output data, or a finished "data product". Figure 1 illustrates this analogy. This analogy has been suggested before in various forms. For example, Huh et al talked of data assembly [Huh, 1990 #572], Levendel applied the analogy to software manufacturing [Levendel, 1991] and Arnold talked of information manufacturing [Arnold, 1992].

We have selected ISO 9000, which is an international standard for product manufacturing [ISO, 1992 #724]. Table 1 shows the main elements of a quality system as proposed by ISO 9000. The standard provides a comprehensive, well-grounded basis for the study of data quality. It is used widely for quality implementation, quality certification and quality auditing. ISO 9000 offers quality management guidance for the supplier's satisfaction (ISO 9004) and quality assurance requirements for the purchaser's satisfaction (ISO 9001, 9002, 9003).

While it has been suggested that this production process framework may foster neglect of important areas such as the corporate data resource view, we believe its advantages outweigh its shortcomings. The main strengths of our approach are that it is a systematic, well-documented and stepwise method.

1.4 Scope of the study

This paper deals not only with data quality in general but emphasizes the management of data quality. One of our aims is to survey data quality research rather than to review current business practices. We define data quality research as the study of models, tools and methods to define, measure, analyze and improve data quality. Data quality is an interdisciplinary problem hence the research reported here is from disciplines such as management science, computer science, economics and organizational studies. Our selection criterion is to include work by authors who explicitly recognize a data quality problem. Nevertheless, we also include other work that we feel is directly relevant.

In Section 2 we use the ISO 9004 guidelines as a framework with which to study data quality concepts, problems and research issues. For each element of ISO 9000 we summarize its associated guideline, survey work from existing data quality research and point out areas unexplored or unresolved by existing data quality literature. Section 3 summarizes our findings and discusses our recommendations for future research.

2 DATA QUALITY MANAGEMENT UNDER ISO 9000

Here we discuss approaches, implications and inadequacies of data quality management assuming it is framed and directed by ISO 9000. For each main element of ISO 9004 (as shown in Table 1), we summarize its guidelines and survey related work from existing data quality research. This section identifies in a systematic manner the key requirements of a data quality system. In particular, we highlight critical aspects of data quality not previously studied in depth or even mentioned by researchers. These areas include the effect of data quality on a product's market position, the problem of how to define customer requirements, the need for a customer feedback system and the question of how to present data quality information to management. Other more obvious issues, such as the need for a data quality metric, are also identified and discussed using our method. Table 2 gives a summary of our analysis.

2.1 Management responsibility and quality system principles

ISO 9000 Guidelines. Responsibility for and commitment to a quality policy belongs to the highest level of management; management should develop and state its policy. The policy should be consistent with other policies and understood, implemented and maintained. Management should define quality objectives, such as performance, costs and losses. Management should establish a quality system -- the organizational structure, responsibilities, procedures and resources for implementing quality management. The quality systems applies to all phases of a product, from initial identification to final satisfaction of requirements.

Management should identify quality factors affecting market position and objectives relative to new products, processes or services in order to allocate company resources on a planned and timely basis. Management should also be responsible for quality manuals and quality plans (quality objectives, procedures, inspection programs etc.).

All elements, aspects and components pertaining to a quality system should be internally audited and evaluated on a regular basis.

Data Quality Research. Although it is widely recognized that responsibility for and commitment to a quality policy belongs to the highest level of management, the definition of quality has not always included data quality. As a corollary to this, we find little research work on management level policies relating to data quality, although some companies have made data quality part of job descriptions [Lindgren, 1991 #394]. The FAA recognized the importance of the quality of federal agency data by incorporating it into their total quality management program [Sparhawk, 1993 #729].

Oman used surveys of data quality as a tool to raise the awareness of the data quality problems of various groups [Oman, 1988 #354]. He sent results back to the people responsible for database quality. Oman's paper focuses on the organizational side of implementing data quality systems.

Researchers such as Davenport maintain that the use of information is often hampered by political or organizational issues [Davenport, 1992 #698]. They believe that there is a pattern to company IS organization which predicts the efficacy of information use. This type of research is complementary to work that demonstrates the tangible impacts of data quality factors.

Open Research. Although some firms have started to tackle data quality problems very few have formalized data quality programs. Few quality systems pertaining to data quality exist to our knowledge (organizations that have implemented data quality systems include the FAA [Sparhawk, 1993 #729], Sun Cos. [Knight, 1992] and AT&T [Redman, 1992]. Significantly, we know of no work that helps management identify data quality factors affecting market position. Researchers also need to consider tangibles and pragmatics such as the form and content of data quality manuals, data quality plans and internal audit procedures.

2.2 Economics

ISO 9000 Guidelines. The impact of quality upon the profit-and-loss statement can be highly significant, particularly in the long term. Quality costs can be broadly divided into operating quality costs and external assurance quality costs. Operating quality costs include prevention and appraisal costs, and failure costs, for example rework, scrap or external failure costs after delivery. External assurance quality costs relate to the demonstration and proof required by customers. Quality costs should be regularly reported to and monitored by management.

Data Quality Research. Cushing, among others (see [Hamlen, 1980 #398; Yu, 1973 #403; Burns, 1975 #396]), developed a tool for auditors in the evaluation of internal control [Cushing, 1974 #397]. He developed a general model of control procedures using parameters of reliability and cost. The model provides a means to compute reliability of a process (probability of completion with no errors). The cost parameters include cost of performing the control procedure, the average cost of searching and correcting for signaled errors and the average cost of an undetected error.

Ballou and Tari presented an optimization heuristic for balancing the costs of parameter estimation and data quality [Ballou, 1989 #503]. Using parameters of cost of undetected errors, stored data error rate, effectiveness of data repair procedure and all associated costs the model uses operations research formulation to allocate available resources in an "optimal" manner. They note in particular the need to obtain better estimates for the penalty costs of poor information quality. This heuristic tackles the problem of whether it is better to identify and correct few, serious errors in one database or more widespread but less severe quality problems in another database. This allows a high level of economic analysis.

Mendelson looked at the costs of incomplete information and their impact on database design [Mendelson, 1986 #704]. Kwan and Rotem made an analysis of the tradeoff between data accuracy and performance of databases [Kwan, 1992 #706]. This identifies a key economic motivation for data quality. Feltham took an accountant's approach to valuing information [Feltham, 1968 #736]. He did this by constructing a payoff function in terms of future events and prediction of those events.

TQM literature also offers many generic guidelines for estimating quality costs of prevention, appraisal and failure see, [Juran, 1989; Oakland, 1989 #544]. These can be applied, with some interpretation, to data quality.

Open Research. No research is known to the authors that deals with the economics of external failure (although anecdotes abound). For some businesses the cost can be catastrophically high. Thus, persuasive examples of the impact of data quality upon the profit-and-loss statement are required. However, we note that Crosby, when looking at product or service quality, estimated that manufacturing companies spend over 25% of sales doing things wrong and service companies spend more than 40% of their operating costs on wasteful practices [Crosby, 1979]. Moreover, studies have demonstrated a connection among quality (albeit not data quality specifically), market share, and return on investment:

Not only does good quality yield a higher ROI for any given market share (among businesses with less than 12% of the market, those with inferior product quality averaged an ROI of 4.5%, those with average product quality an ROI of 10.4%, and those with superior product quality an ROI of 17.4%); it also leads directly to market share gains. Those businesses... that improved in quality during the 1970s increased their market share five to six times faster than those that declined -- and three times faster than those whose quality remained unchanged [Garvin, 1983]

To our knowledge, no work has attempted to estimate costs of data quality assurance. Another strand of future research is how to interpret and present data quality information to management.

2.3 Quality in marketing

ISO 9000 Guidelines. The marketing function should accurately define the customer requirements, including an assessment of any unstated expectations or biases held by customers. A product brief should translate customer requirements into product specifications such as performance characteristics, tolerances, packaging, quality assurance etc. Marketing should establish a customer feedback system.

Data Quality Research. Much research has focused on customer requirements of information systems and information quality. In contrast, little research has addressed the translation process from data customer requirements into product specifications.

Delone and McLean (1992) listed information quality and user satisfaction as two of the six major dimensions for evaluating the success of information systems. Their review of the literature found more than 20 measures of information quality, including the typical ones mentioned above as well as precision, conciseness, usefulness, content and understandability. Availability, traceability, and credibility (Loebl, 1990) have also been cited, among many others, in the literature. Other research on evaluating information systems from the user's view generally includes some "quantitative" data quality attributes. For example, research to define and develop instruments to measure user satisfaction with information and information systems cited attributes such as accuracy, timeliness, precision, as well as softer dimensions such as reliability, completeness, and relevancy, e.g., (Bailey and Pearson, 1983; Ives, Olsen, and Baroudi, 1983; Kriebel, 1979). Kim supposed that there are three different perspectives: user satisfaction in terms of attitudes towards MIS; user satisfaction in terms of information quality; and user satisfaction in terms of MIS effectiveness [Kim, 1989 #735]. He offered a synthesis of these three perspectives.

Zmud conducted a survey to establish important user requirements of data quality; the analysis showed some of the user intuitions of the dimensions of data quality [Zmud, 1978 #718]. Similar work was done by Baroudi, who described a measure of user satisfaction of information [Baroudi, 1988 #582]. A useful marketing approach would be similar to Delone and McLean's study (1992) on measures for evaluating the success of information systems.

Wang's methodology for data quality requirements modeling is noteworthy in that it tackles the translation process [Wang, 1993 #531]

Open Research. More work is required on how to gather and translate customer requirements (especially unstated expectations) into "hard" design specifications. Similarly, no thought has been given on how to establish a customer feedback system for data quality or how to accurately define customer requirements.

2.4 Quality in specification and design

ISO 9000 Guidelines. The specification and design should be such that the product or service is producible, verifiable and controllable under the proposed production, installation, commissioning or operational conditions. The quality aspects of the design should be unambiguous and adequately define characteristics important to quality, such as acceptance and rejection criteria. Tests and design reviews should be conducted to verify the design.

Data Quality Research. A key area of data quality research is how to express the quality aspects of the design in an unambiguous and rigorous form. Most researchers have an intuitive grasp of data quality but few have agreed on the type and nature of data quality problems. Some researchers have dealt exclusively with quality problems of one type: errors. Morey, for example, constructed a method to estimate the "true" stored error rate (an interaction between to incoming quality problems, introduced quality problems and the processing) [Morey, 1982 #402]. Others have recognized the multi-faceted nature of data quality: data can also be defective by virtue of their age (i.e. out-of-date), inconsistencies in cross organizational definitions, software defects, communication errors, hardware errors (for example, [Redman, 1992 #682]). This multi-faceted nature of data quality is referred to as data quality dimensions. A dimension is an independent or orthogonal aspect of data quality. For example, timeliness is often considered as one dimension and consistency another. Data may have quality problems on one dimension while being satisfactory along others. The use of dimensions is helpful for classifying and understanding quality problems and has been used before in other disciplines, see [Garvin, 1987 #372; Zeithaml, 1987 #349].

Ballou and Pazer (1985) defined accuracy as when "the recorded value is in conformity with the actual value." Accuracy, defined in terms of the frequency, size, and distribution of errors in data, has been the key dimension used in studying data quality in accounting and auditing, e.g., [Bodnar, 1975; Cushing, 1974; Johnson, 1981 #401; Knechel, 1983 #679; Knechel, 1985 #680; Yu, 1973 #403]. In addition to defining accuracy, Ballou and Pazer defined timeliness (the recorded value is not out of date), completeness (all values for variables are recorded), and consistency (the representation of the data value is the same in all cases) as key dimensions of data quality [Ballou, 1985 #502]. The team from AT&T Bell Labs [Huh, 1990 #572; Redman, 1992 #682] described data quality dimensions from a conceptual and practical viewpoint. They introduced 23 dimensions of data quality, including accuracy, completeness, consistency, and currency/cycle time.

Paradice and Fuerst formulated a measure of the error rate of stored MIS records [Paradice, 1991]. They used this as a benchmark for comparing actual error rates with the theoretically smallest attainable error rates and suggested a method for assessing an organization's data quality. Also, they offer guidelines for deriving values for parameters of their data quality model.

Jang et al. developed a data quality reasoner that provides an automated form of judging data quality [Jang, 1992 #552]. The reasoner allows an objective measure of data quality. Wang, Kon and Madnick proposed a methodology for specifying data quality requirements in the context of enterprise data modeling [Wang, 1993 #531]. Work to formalize concepts such as quality attributes (versus entity attributes), essential in translation of data customer requirements into product requirements, has been initiated by Storey [Storey, 1993 #738]. These efforts go towards providing a disciplined basis for specifying data quality attributes. Slissenko provides a theoretical backgrounder in the measurement and effectiveness of computer knowledge processing [Slissenko, 1991].

Open Research. A "standard" metric or acceptance test is required which could be used at the design stage or during the production process.

2.5 Quality in procurement

ISO 9000 Guidelines. The procurement of supplies (data) should be planned and controlled. A procurement quality program should include the selection of qualified suppliers, the agreement on quality assurance, the agreement on verification methods and receiving controls and records.

Data Quality Research. To the authors' knowledge no data quality research has tackled the area of data procurement. Our experience with information vendors and users suggests that data feeds are generally perceived as reliable, correct sources. Organizations clearly need procurement guidelines; the guidelines should include procedures that certify that data received conforms to specified requirements.

Open Research. There is substantial scope for research on procurement guidelines (note the EDI community faces the same problems), particularly the criteria for the selection of qualified suppliers.

2.6 Quality in production

ISO 9000 Guidelines. Planning of production operations should ensure that these proceed under controlled conditions in the specified manner and sequence. Verification of the quality status of a product, process, software, material etc. should be considered at important points in the production sequence. The verifications should relate directly to finished product specifications. Documented test and inspection procedures should be maintained. New methods for improving quality and process capability should be encouraged.

Data Quality Research. Redman and a team of researchers at AT&T's Bell Labs developed a technique called data tracking [Huh, 1990 #572; Pautke, 1990; Huh, 1992 #728]. They used a combination of statistical control and manual, or otherwise, identification of errors and their sources. This TQM-inspired model is practical and comprehensive but does not attempt to tackle economic or organizational impacts. Underpinning data tracking is the idea that processes which create data are often highly redundant. Data tracking uses the redundancy to determine pairs of steps in the overall process that yield inconsistent data. Changes that arise during data tracking are classified as normalization, translation and spurious-operational. Spurious-operational changes occur when fields are changed during one sub-process; they indicate an error somewhere in the process. This allows the cause of errors to be systematically located. The team's approach also tackles the requirement of documentation and test procedures.

Undoubtedly ad hoc production methods exist, for example, to purge redundant addresses, but these have not been reported.

Open Research. Verification and metrics both underpin techniques for production. In addition to these areas, we suggest that the study of process, data flow and human or automated procedures in the context of quality management need to be addressed further.

2.7 Control of production

ISO 9000 Guidelines. All materials and parts should conform to appropriate specifications and quality standards before being introduced into production. Special consideration should be given to shelf-life and deterioration control. Where in-plant traceability is important to quality, appropriate identification should be maintained throughout the production process to ensure traceability to original material identification and quality status. All equipment should be proved for bias and recalibrated or verified regularly. A product should be evaluated after any process change to verify that the change instituted had the desired effect upon product quality.

Verification status of materials and assemblies should be identified throughout production. The identifications, in the form of stamps or tags, should distinguish between verified and unverified material. Provision should be made for positive identification and control of all non-conforming material.

Data Quality Research. Wang et al. [Wang, 1992 #553] developed a technology framework of data tagging to provide additional attributes of data, or metadata, to answers the user need for tracing data source. They defined a data quality parameter as a qualitative or subjective dimension of data that a user defines as important (for example, timeliness). Their data quality indicator is a data attribute that provides objective information about the data (such as, source, time, collection method). They also introduced the notion of a quality database management system (QDBMS), one that supports data quality related capabilities (for example, an SQL that includes a facility for data quality indicators).

Open Research. There is a need for a mechanism or tool to manage data quality shelf-life (out-of-date data) and deterioration control (data corruption). Moreover, a method for positive identification and control of all non-conforming material (poor quality data) is required. Methods for "recalibration" of measuring instruments (that is, specialized software) are also required.

2.8 Product verification

ISO 9000 Guidelines. In-process inspection or tests should be considered at appropriate points in the process to verify conformity. Two forms of final product verification may be used: acceptance inspections or tests (for example, 100% screening) and product quality auditing of sample units. Deficiencies or deviations should be reported and reworked or repaired.

Data Quality Research. Redman's data tracking technique offers a framework for production verification (see also Section 2.6). [Huh, 1990 #572; Pautke, 1990; Huh, 1992 #728] Product verification hinges on quality metrics (see Section 2.4) Survey data techniques such as imputation could be applied to rework data [Zarkovich, 1966 #505].

Open Research. Little work is known that deals with acceptance inspections or the reporting of deviations in data elements or database records. To our knowledge, no researcher has synthesized methods for data verification and repair in multi-type database systems.

2.9 Control of measuring and test equipment

ISO 9000 Guidelines. Sufficient control should be maintained over all measurement systems used in the development, manufacture, installation and servicing of a product. Control should be exercised over instruments, sensors and related software. Correction to out of control measuring and test equipment should be performed.

Data Quality Research. In the realm of data quality a measurement system presumably would be implemented as software. Thus this measurement system would be subject to normal software quality assurance schemes.

Open Research. We believe the field of software quality, verification and validation covers these guidelines.

2.10 Nonconformity

ISO 9000 Guidelines. Suspected non-conforming items should be immediately identified and the occurrences recorded. They should be segregated from conforming items. Non-conforming items should be subject to review by designated persons.

Data Quality Research. Segregation in the realm of data quality can be achieved by marking non-conforming data. These records could be extracted and placed in a separate database or file. To our knowledge, no formal research on how to segregate and review non-conforming data records.

Open Research. There is some scope for analyzing what to do with non-conforming data and creating techniques to aid in reviewing defective data.

2.11 Corrective action

ISO 9000 Guidelines. The significance of a problem affecting quality should be evaluated in terms of its potential impact on such aspects as production costs, quality costs, customer satisfaction etc. The relationship of cause and effect should be determined. Preventative action should be initiated. It may also be necessary to revise the procedures used to detect and eliminate problems.

Data Quality Research. Ballou, with Pazer, [Ballou, 1985 #502] described a model that produces expressions for the magnitudes of errors for selected, terminal outputs. It was intended for analysts to compare alternative quality control strategies. It is useful in assessing the impact of errors in existing systems. Ballou also developed an operations research model for analyzing the effect and efficacy of using data quality control blocks [Ballou, 1987 #571].

Morey described mathematical and practical ways to improve data quality in existing systems [Morey, 1982 #402]. Because Morey's model incorporates an estimate of transaction time and its effect on timeliness problems it is able to show how, for example, varying the minimum batch processing time can effect the stored error rate. Morey also suggested tightening edits to reduce the statistical Type I and Type II errors. Cushing suggests several possible control measures including procedures to review and reconcile individual remittances prior to posting and separation of the function of accounts receivable clerk from that of cashier [Cushing, 1974 #397].

Redman noted that improvements require resources as well as continued monitoring of the whole data production process [Redman, 1992 #682]. He suggested that accuracy problems may be corrected by improved representation schemes and timeliness problems by identifying and improving queues or transmission areas in the IS; system redesign, to provide parallel data flow, may also reduce data flow times.

Open Research. Although much has been done on impacts and cause and effect relationships there is still a need for work on how to take (or automate) preventative action. Likewise, the link between poor data quality problems and revision of the procedures used to detect and eliminate problems has not been tackled.

2.12 Handling and post-production functions

ISO 9000 Guidelines. The marking and labeling of materials should be durable and remain intact from initial receipt to final deliver. Items with limited shelf-life should be clearly identified.

Data Quality Research. We cannot see a direct analogy between handling and a corresponding activity in the data product realm because information systems do not usually involve any "handling" of their finished products.

Open Research. A possible topic is to establish the analogy.

2.13 Quality documentation and records

ISO 9000 Guidelines. The quality management system should establish, and require the maintenance of, a means of identification, collection, indexing, storage and retrieval of quality documentation and records. Information maintained includes inspection reports, audit reports, test data and calibration data etc.

Data Quality Research. Data dictionary techniques allow a central repository of data definitions, change information etc. However, these systems rarely include explicit facilities for quality information. Wang's scheme of data tagging supplies a limited form of quality documentation [Wang, 1992 #553].

Open Research. A thorough examination of the need, role and implementation strategies of data quality documentation is required.

2.14 Personnel

ISO 9000 Guidelines. Personnel requirements fall into three categories: training, qualification (experience and formal qualification) and motivation.

Efforts should be made to motivate quality of performance and awareness of quality. Management should measure quality achievement and provide related recognition when achieved.

Data Quality Research. Oman's organizational awareness method apparently improved data quality (this could have been a Hawthorne effect) by attacking a motivation and awareness problem [Oman, 1988 #354]. We know of no other research on personnel issues relating directly to data quality as opposed to product or manufacturing quality.

Open Research. We do not see a pressing need for research in this area as it is likely that existing TQM techniques for personnel could be applied.

2.15 Product safety and liability

ISO 9000 Guidelines. Safety aspects should be identified to enhance product safety and minimize product liability. Relevant safety standards should be identified and applied. Manuals and labeling should be analyzed to minimize misinterpretation. A traceability system to facilitate product recall should be instigated.

Data Quality Research. Apart from Wang's work on data tagging [Wang, 1992 #553] no work is known on "safety" or liability limitation. Liability when false information is delivered is potentially a serious problem. With the increasing use of information "highways" there may be a need for data safety. Note that the Commission of the EC published a proposal for a Council Directive concerning the protection of individuals in relation to the processing of personal data [Chalton, 1991 #734]. The Directive contains chapters on rights of data subjects, data quality, and provisions relating to liability and sanctions. Wright looked at the need in EDI to legally prove the origin and content of an electronic message [Wright, 1992].

Open Research. We recommend an investigation into legal aspects of poor data quality and related liability. Similarly, there is a requirement for methods on how to avoid liability issues. Another area for research is in secure methods for tracing data accessed through remote networks or databases.

2.16 Use of statistical methods

ISO 9000 Guidelines. Statistical methods may be applied in market analysis, product design, reliability prediction, process control, determination of quality levels and, defect analysis/ performance analysis.

Data Quality Research. The TQM literature offers a full suite of statistical techniques that can be applied, more or less, wholesale to data quality. Researchers who have used SPC have generally dealt simply with percentage of incorrect errors [Redman, 1992].

Open Research. Issues of quality definition and problems with non-numeric or highly complex data need to be tackled. Moreover, there is scope for the application of statistics to data product design, reliability prediction and market analysis.

3. CONCLUDING REMARKS

We have learned from our survey of data quality that although much work has been done on the subject, the knowledge base is very fragmented. Several data quality problems have been tackled explicitly or implicitly under the umbrella of database integrity, communications engineering, accounting and other fields. On the other hand, there is still a lack of applied frameworks, tools and case studies for data quality. TQM has virtually ignored the aspect of data quality. Yet, there is clearly a latent but burgeoning recognition of the field of data quality.

Our review, using the data manufacturing analogy and ISO 9000, of the existing data quality literature suggests a number of topics for further research. For example, there is a clear need for techniques that help management identify data quality factors affecting market position. This is a competitive advantage issue, fundamental to business performance. Similarly, the economics of external (data quality) failure and the complementary costs of quality assurance need to be evaluated. Without a model for these costs it will be difficult for management to make a decision on whether to implement data quality management procedures. Related to the market position problem is how to define customer requirements. Moreover, it would be useful to establish a customer feedback system for data quality. There is also a need to study the link between poor data quality problems and revision of the procedures used to detect and eliminate problems.

Researchers also need to investigate how to present data quality information to management and how it should be interpreted by business leaders. Managers will also want to know how to select suppliers of information.

A fundamental technical need is a data quality metric and a way to express the data quality aspects of a product design in an unambiguous and rigorous form. Without this underpinning there can be no way to develop the next step -- data quality verification and test procedures. These are tied to a method for the positive identification and control of all poor quality data. Control could be achieved with the aid of statistical techniques, once it is understood how to apply them when source data are not always numeric data. Given a quality metric it may be straightforward to develop a mechanism or tool to manage data quality shelf-life (out-of-date data) and deterioration control (data corruption).

Its widespread acceptance indicates that ISO 9000 provides a systematic, extensive view of what is required for a quality management system. Our use of ISO 9000 coupled to the data manufacturing concept has helped us to broaden our literature search and given us a better understanding of some of the unexplored, critical areas of data quality management. We believe we have made some major steps towards the integration of the dispersed data quality literature. This paper provides a useful summary to the new data quality researcher and offers evidence that data quality management is recognized increasingly as an important branch of the quality management discipline.