by Sophie Allen
MasterSoft Research Pty Ltd, Sydney Australia
In today's competitive business environment corporations in the private sector are being driven to focus on their customers in order to maintain and expand their market share. In the public sector the de-regulation and privatization descending upon government departments is also pushing these traditional monopolies into competition. Suddenly, telephone and power utilities are scrambling to discover their customers. This shift is resulting in customer data and information about customers being viewed as a corporate asset. As with any asset, it has certain attributes and value: Who are our customers? What is the profile of our customers? What are our customers individual needs? What is the profile of our most profitable customers? To be able to accurately answer these questions, an organization must be equipped to analyze their customer data, and use the data to derive information.
Historically, customer data has been gathered by separate business units within an organization to fulfill government regulations or as a means to send a piece of mail. This data is generally not of sufficient quality to derive answers to the types of questions organizations want to answer. Information technology (IT) departments are therefore being asked to delve into this new data management arena. They are setting out to improve customer data quality and effectively manage customer data. The challenges faced to achieve these goals are the ability to:
Customer data consists of many components with Name & Address being the key. A person or companies Name & Address is THE identifier of a customer. Organizations may use: policy, bank account, or even telephone numbers but these really only apply to a particular line of the business. In telecommunications, for example, a customer may have a series of numbers: mobile, residential, business, calling card etc.
Name & Address as a data type is very difficult to deal with and manage. This data is volatile... customers come and go, addresses change, names change. This data is often cluttered when entered. Name & Address fields on data entry screens are usually free format and ripe for use to enter comments without any edits. Name & Address is subjective... it can be written in a number of different ways and still be the same. Finally, there is no application independent standard for Name & Address that can be used as a measure of quality.
Name & Address data quality needs to be measured at
two levels: the quality of individual Name & Address components,
and the integrity of the customer base as a whole. At the individual
level metrics include the degree of: structure, content, completeness,
consistency and accuracy. While the integrity of the whole customer
base can be measured by examining the accuracy and completeness of relationships
identified: between individual customers and between customers and the
The Name and Address Data Integrity Software, (NADIS) is a suite of software products developed by MasterSoft Research Pty Ltd in Sydney, Australia. It was specifically developed to assist organizations in the management of Name and Address. Two expert systems which parse and match data are at the core of the software. These rule-based systems or engines are implemented with transition table technology. The engines operate on a variety of platforms and are packaged into three products: ScrubMaster, SearchMaster and Onlooker. ScrubMaster is used parse files of Names and Addresses. SearchMaster is used to identify relationships within and across files. Onlooker incorporates both engines for the on-going protection and management of clean databases.
The parsing engine converts Names & Addresses to a common, structured and standardized format. The application independent common format developed by MasterSoft is called Universal Name and Address (UNA). The parsing engine also reports on and corrects data inconsistencies. Among the data inconsistencies identified are missing or extraneous data, invalid characters or illogical information.
The matching engine identifies relationships between parsed
Names and Addresses. It emulates the same reasoning we use when comparing
the differences and similarities between Names and Addresses. The matching
engine accounts for the common typographical, transliteration, and elision
errors made when keying data.
Corporations are becoming customer focused because of competition. In the private sector, competitive pressure is forcing corporations to know and get closer to their customers. They are targeting channel marketing efforts to the individual rather than rely on mass marketing mechanisms. This same shift is emerging in the public sector where the de-regulation and privatization of traditional government monopolies is pushing these organizations to discover their customers.
At a strategic level corporations are seeking to develop the corporate - customer relationship to increase revenue while minimizing the cost of sales, and to maintain or increase market share. At an operational level, these strategies are being implemented with the following objectives:
Take a telecommunication company's initiative to improve
customer service by setting up a one-stop service center. A customer will
be able to ring one main number and the customer service agent is equipped
to handle ALL the customer's queries. The customer service agent must have
a picture of all the customer's dealings with the corporation. The ability
to provide this level of service hinges on having customer focused data.
Traditionally, organizations have used internal numbers to relate with, and identify, customers. From an IT point of view numbers were great unique identifiers of items such as insurance policies, bank accounts, passports, trust accounts, and transactions. Product systems and lines-of-business were built around these numbers. Of course, all interaction between the customer and the company revolves around the number. The problem is that the same customer may have a series of identifiers. In most systems, a user could not search the database to find a customer by their name.
Data about the customer was only obtained if the organization needed to send mail to the customer or for reporting requirements to government departments. In a bank a mailing address for a credit card is necessary to send a statement where as a passbook account may not need an address. This type of data was not the focus of the system, but merely collected and maintained.
MasterSoft has assisted many organizations in making the shift from having customer data to having customer information. These customer data quality projects typically start with the following questions:
Typically, a line of business can enumerate the number of policies, accounts, agents but it can only guess how many customers that translates to. Expanding the scope across the corporation usually yields a shrug of the shoulders.
Agreement on who is a customer is another difficult proposition. If a customer last bought a product or service 2 years ago, are they still a customer? Is the name in a care of address a customer? Are agents who act on behalf of others customers? Is the beneficiary of a policy a customer?
Which of all the customers are the most profitable? What areas do they live or work in? How many households does this comprise? How many of these customers live in the same household?
Before this information can be extracted from the customer
base, the organization must measure and improve the quality of its customer
Turning the customer data in these legacy systems into customer information as the focal point of systems poses significant challenges to the enterprise:
Agreement from Lines of Business - To fully exploit customer as a corporate asset, all lines-of-business need to participate and contribute to a strategy. This commitment is frequently difficult to obtain because each line of business has its own goals and accountabilities. For customer focus to be profitable it must be embraced by all levels of the organization and initiated by the very top.
Organization Structure - Few organizations, have a structure chart with the title Client Data Manager, Client Data Owner or Data Custodian. The enterprise must recognize and allocate these distinct responsibilities. With systems built along product lines or lines-of-business, these three titles (if they existed) were within that area. If client data is a corporate asset, resolving client data ownership and management is not an insignificant task. A recent survey conducted by our North American distributor, Group 1 Software, reinforced this point. Of the 100 responses from senior IT executives, there where 50 different answers to the question who owns the data ?
Cost-Justification - Which departmental budget
absorbs the costs to measure, improve and maintain customer data quality
to meet the operational and strategic objectives? It is difficult to tangibly
cost-justify the direct value of this corporate asset, and data quality
projects need resources from across the company.
One common element is that IT departments are charged with the task of physically providing information, data quality is a new arena. Over the years, IT departments concentrated their efforts in the 70ís on hardware, in the 80ís on software and in the 90ís are moving towards data.
Even today, agendas of many Data Warehouse conferences do not include seminars on Data Quality. Topics include meta data, databases, data warehouse architecture and technology, decision support systems, principles. The EXTRACT, TRANSFORM and LOAD phase of the process seems to be always glossed over as if it just happens. This phase consumes a lot of effort and is not a sexy proposition. However, failure to appropriately address data quality issues will certainly guarantee the failure of Data Warehouse projects.
The skills required to effectively manage data quality
are typically not available in IT departments. Most application development
project teams include project managers, analysts, programmers, and business
analysts. It is rare that skills in managing data for the corporation are
included in the project team. As a software development house, MasterSoft
finds it difficult to locate staff with experience in data quality. It
is difficult to hire people who know answers to questions such as: What
are the issues involved? What are the pitfalls? Where do we start? How
and what do we measure? How do we go about all of this on a scale of millions
Name & Address is the data which is the only common identifier of a customer. Even though it is key to identifying every customer, it is subjective and volatile and therefore difficult to manage. This view was supported in the same survey conducted by Group 1 Software in which Name & Address had the highest response (31%) to the following question, Of all the legacy customer data you've collected, what type of data presents the most problems when it comes to ensuring its integrity?
The properties which make Name & Address difficult include:
Subjective - People can write the same Name & Address data in many different ways. When we interpret data, we account for differences by applying knowledge about known cultural norms in the way Name & Address data is written. This is easy when a person physically reviews and compares two names and addresses but it becomes an onerous task when having to automatically apply these norms to millions of Names & Addresses. The problem is further compounded by the different ethnic backgrounds of data in a global market.
Free Form- Name & Address fields in data entry screens are typically not structured. Being perhaps the only free form fields available, they are ripe for entry of comments, account numbers, statuses etc. Usually, anything that doesn't fit into another field is jammed into Name & Address fields. Also, Name & Address is very difficult to edit for completeness, consistency or accuracy. So, Name and Address is just added or updated regardless of quality.
Volatile - Names & Addresses change all the time; they are not static. This volatility is caused by corporate mergers, marriage name changes, movement of residential addresses, individual customer preferences - in communication, privacy of activity, changing residential address, changing postal addresses etc. This volatility needs to be managed.
Standards - Using conformance to a data standard
as a measure of quality is difficult because there is no independent
data standard for Name & Address either internal or external. Historical
use of Name & Address has been for particular purposes - mailing or
government requirements. A company conforms to these standards because
of enforcement by the receiving agency. For example, when a bill mailing
run is performed, data is extracted from the database and massaged (if
possible) to meet the local mailing standards before addresses are affixed
to the envelopes. The massaged customer data held by the organization is
typically not updated. The same data may be extracted and massaged differently
to meet another standard. The quality of the data held by the organization
is not improved.
A definition of customer data quality that MasterSoft has developed based on experience with our customer base is: The extent to which customer information remains reliable and consistent across the corporation . Unfortunately, most corporations seem to view data redundancy or duplicate records as a measure of Name & Address quality. The reality is that this is just one facet of customer information quality. Name & Address Data quality needs to be measured at two levels:
Possessing high quality individual Name & Address
data is an essential foundation to deriving reliable information about
Before an organization can extract reliable information from Name & Address data as a whole, it needs to identify, validate and measure the quality of the individual Names & Addresses. This is the necessary first step. The following facets of Name & Address can be used to measure the quality of individual Names & Addresses across the entire customer base.
Structure - Does the Name & Address have the appropriate structure necessary to meet all the organization's needs? Is the data completely structured? Are the key elements structured? Free format data does not allow the organization to effectively communicate with the customer in a wide variety of ways or perform market segmentation.
Completeness - What are the key elements for minimum name data? Is MR JONES acceptable? Will it meet the organization's needs? What are the key elements for minimum address data required? Is an address NEW YORK, NY acceptable? Will it meet the organization's needs? What other elements are missing? Determining the missing elements, can only be undertaken if the data is structured. If the data is structured and the elements are missing, are they required?
Consistency - Has the same spelling been used to reflect the same meaning? Does the Name & Address conform to determined standard spellings for commonly used terms? Do a customer's initials appear on one system but a full name on another?
Accuracy - Is the data accurate? Is the name spelt
correctly? Is the address current, or has the person moved? Is the Name
& Address reliable?
Once the quality of the individual Names & Addresses is measured and improved to the level required, the Names & Addresses can be used to reliably :
MasterSoft's experience with these three objectives is to attack them in the order listed. Duplicates need to be identified or removed, otherwise effort is wasted on deriving other data for Names & Addresses which may be eliminated. Once these steps are completed, the data can be used to extract reliable information.
The information required of the customer data typically falls into two categories of relationships or links:
Between a customer and the organization- For a particular customer or group of customers it is important to have a view of all dealings the customer has with the organization as a whole. A bank, for example, may want to determine all banking products a customer has (either solely or jointly).
Between a customer and other customers- For a particular customer or group of customers it is important to have a view of all relationships the customer has with other customers. Examples in banking include:
The quality of this derived information can be assessed based on:
Customer data quality projects do not end once the quality
is assessed and improved. They also do not end when information is derived.
To maintain the quality and the information, the organization must implement
an on-going strategy for customer data management.
MasterSoft International was founded in Sydney in 1989
to develop knowledge-based Name & Address matching software for the
Australian market. The company has expanded operations into New Zealand,
and appointed distributors in the USA, and Canada. To further increase
its level of customer service MasterSoft International divided its operations
in 1995 to form MasterSoft Research Pty Ltd, also headquartered in Sydney.
MasterSoft Research is solely responsible for developing NADIS (Name and
Address Data Integrity Software) products for vertical and global markets
and to undertake joint research projects. Over the last six years MasterSoft
has grown dramatically and is now generating revenue of over AUD $6 million
with 22 employees. NADIS is used world wide by over 70 customers ranging
from government departments to large insurance and banking corporations.
Two case studies of NADIS customers are included in the appendix which
describe how NADIS has been used to manage Name and Address Data Quality.
NADIS incorporates MasterSoft's Name & Address processing products, ScrubMaster, SearchMaster and Onlooker. These products use sophisticated inference engines, which emulate the processes used by people to assess names and addresses, locate relationships between client records despite keying errors, spelling discrepancies and differences in data format.
ScrubMaster: structures and standardizes a company's files of Name & Address data. The software identifies every element in a Name & Address record, and then converts these elements to the Universal Name and Address (UNA) data standard. UNA is NADIS' standard format for describing Name & Address data. The UNA is critical to customer information system (CIS) applications for two reasons. First, it provides a uniform standard so users can effectively manage and model their customer Name & Address data. Second, it enables users to easily move "cleaned" data between different computer systems and applications.
SearchMaster: matches files of Name & Address data. The software processes the output of ScrubMaster to analyze all relationships between Name & Address records. It emulates the inferences used by people, accounting for such variables as spelling and phonetic differences to provide reliable and accurate matches on name or address, or Name & Address combined.
The result is much cleaner names and addresses because the information has undergone much tougher scrutiny than less sophisticated software can provide.
Onlooker: is an on-line package combining Name & Address scrubbing, matching and search key generation. It can be installed as a batch-driven system or as a true interactive system. Onlooker uses the same scrub and match engines used in other MasterSoft products.
Competition is forcing corporations to get to know their customers. This poses a number of challenges for corporations and their IT departments to switch from customer data to customer information. IT departments are now focusing on data quality which is a new arena,
Name & Address is the key data component of customer data yet because of its nature it is difficult to manage. Frequently, data quality of Name & Address is mistakenly equated to duplication. To properly assess Name & Address Data Quality, a corporation needs to identify and measure the quality of individual Names & Addresses before the integrity of the Name & Address data as a whole. This approach will provide the corporation with the reliable information it requires about its customers to achieve related business objectives.
NADIS, a suite of Name & Address Data Integrity products developed by MasterSoft Research, is used by organizations world wide to measure and improve the quality of names and addresses of customers.
This case study is taken from a DEETís (Australian Federal Department of Education, Employment and Training) presentation at the MasterSoft User Group in 1995. The presentation was prepared by:
Australia-wide the Department of Education, Employment
and Training has over 16,000 staff and 676 offices administering a wide
range of funding programs that are supported mainly by computerized application
systems and occasionally by clerical support. The Department's six million
plus clients include jobseekers, labour market program participants, Australian
and over seas students, educational institutions, employers and other commonwealth
and state government departments. There are 10 major IT systems and a further
70 smaller systems at DEET running on a diverse range of hardware and software
platforms. In the past few years, interfaces have been created between
all these systems.
An ongoing need to support new government initiatives means DEETís applications are constantly being upgraded. Because of the complexity of the hardware and inflexibility of many of the applications, heavy demands are placed on system maintenance and development. This has created a number of problems including duplicated client data. The large number of duplicate clients on DEETís systems has resulted in:
In mid 1993, a massive clean up exercise was carried out to remove duplicates. This succeeded in reducing the number of duplicates by over 60 percent. But within 6 months the amount of duplicates rose again to the original level. The department looked for a more constructive and permanent solution, such as putting in place preventative measures at the points of data entry. It was not possible to plug all data entry points simultaneously in 80 plus systems. Since reducing the risk of fraud and duplicate payments is a high priority within the department, they first focused on its financial system.
The seven systems used to make DEET payments hold over
900,000 client records. Prior to introducing preventative measures across
these seven systems, DEET developed a common standard for capturing and
storing the data. Before implementing preventative action, DEET created
a weekly average of 560 duplicates. This compares to 75 duplicates per
week which can be easily detected an removed after the on-line preventative
measure was implemented.
Among DEETís future plans:
This case study was provided by MasterSoft for incorporation
into a Data Quality Report co-authored by the Cambridge Research Group
and Cambridge Market Intelligence. It covers a relatively unique application
of Data Quality to help solve common global law enforcement problems.
Formed by the Australian Federal Government in 1989, the Australian Transaction Reports and Analysis Center (AUSTRAC) assists in the deterrence and detection of tax evasion and criminal activity, money laundering from organized and corporate crime, and drug trafficking.
AUSTRAC collects, analyzes, monitors and disseminates
transaction report information from domestic and international financial
institutions and other cash dealers, each of which is required to report
significant cash transactions. The types of organizations range for small-time
bookmakers to the 5,500 branches of the four major Australian banks. The
latter contribute four fifths of the reporting volumes. Reports produced
by AUSTRAC are made available to the Australian Taxation Office (internal
revenue) and law enforcement agencies. The Center's annual budget is around
US $7.5 million, of which 60% goes to IT.
Over 7,000 organizations report electronically to AUSTRAC.
Each of these has its own format for Name & Address data, so a key
issue for AUSTRAC was how to cost-effectively and rapidly standardize and
monitor this incoming data. Because of the sheer variety of formats the
challenge was to find an automated system solution which could consolidate
and de-duplicated this demographic data.
AUSTRAC requirements for Data Quality management tools:
To use their own words, a solution that offered AUSTRAC a balance of completeness, accuracy, adequacy and timeliness .
The system was developed for a Tandem Cyclone, running
non-Stop SQL under Guardian. The AUSTRAC application itself was written
in COBOL , and the overall database contains more than 5 million reports,
with annual growth expected at 3 million reports. Including the AUSTRAC
in-house user base about 1500 tax and law enforcement officers have on-line
access to AUSTRAC data.
The reports provided to AUSTRAC have yielded significant results in the short time that the agency has been operational. The use of a specialized Data Quality tool has improved the solution in three main ways:
The data processing requirements of AUSTRAC reflect those of many commercial systems, but far exceed them in complexity and variety. The ability of the NADIS product to satisfy the AUSTRAC requirements was an example of wise management decision to purchase a product that embeds a high degree of specialization and expertise. Although AUSTRAC probably did not think in terms of building a data quality system per se, without a data quality tool their requirements would have been very difficult to meet so quickly and the project itself could easily have become unmanageable.