Business Dynamics Change - Company expands to new markets.
- Company purchases another company and consolidates varied set of disjoint applications. As the business integrations for acquired or merged businesses is bound by deadlines, many short-cuts are made to 'make it happen', while taking known or unknown risks to the Data quality.
- External requirements are received like new regulatory report OR change in financial reporting by parent company. These urgent needs lead to manual work-around which if continued for long result in data quality issues.
TIP- The above examples are typical of an organizational event, which is the root-cause for data quality issues. Smart organizations will pro-actively have the processes related to data and BI strategy for events like Geographical Expansion, mergers and acquisitions. Deploy a certain band-width of an organization build these processes. The reason for this advance preparation is that when an event of acquisition, merger OR expansion happens, there is a tremendous time pressure.
Control OR De-control of Applications OR Databases
- Due to cost-time issues with core applications, business units create their own set of local applications with OR without the knowledge of IT. These applications do not adhere to standards of data, data model OR interfaces.
- Users sometimes copy the databases of the local applications into their desktops. Many a times, an application (which could be excel based OR on MS access) is running in different desktops with their own stand-alone versions. In this scenario, one looses control on versions as well as the standards.
- Databases of these applications are not maintained by IT and therefore their back-ups and also their administration is not done leading to loss OR overwriting of data.
- The applications, as they undergo organic development do not have a proper version. If wrong versions are placed (both application and database scripts), it leads to data issues.
- Typically, de-controlled applications don’t use central metadata OR data model, their business rules and models could be mismatched and so could be their data.
Organizations have tried unsuccessfully to have all the business critical automation to be owned by IT. The business owned small-time applications keep on mushrooming as IT can never fulfill all the demands of business. Moreover business typically feels that small applications can be done more cheaply by them than by IT.
TIP- Both the extremes of fight or flight are risky to manage this root cause of data quality issues. IT somewhere needs to take the reality check that the small time business applications are inevitable. The idea should be to manage their road-map in way that they cause least disruption to the IT land-scape. The CIO can work with business to Inventorize, classify and road-map the business owned applications and take the prudent choices.
Application Evolution leading to Data Quality Issues
- As an application evolves, its database also undergoes a design change. This leads to some new fields, where data for historical transactions is not recorded. This may not impact the transactions, , but can impact the management information.
- As an application evolves, new business rules are created. Many a times those business rules are not applicable to historical transactions. This creates issues, when you are analyzing the information across current and future information.
System Work-Around
- Sometimes due to a field not available, an alternate field is used, which leads to misinterpretation at later stage, and the data is nearly lost after the initial set of users leave. For example- Data entry person entering the alternate address of the customer in 'comments' field.
TIP- Typically these kinds of fields are descriptive, and they have mainly reporting implications. They are not used for any kind of processing. Adding these fields in the database and in data entry forms (&screens) should not cost much. These kind of data quality issues should be addressed ASAP due their low-cost and the data quality mess they can create.
Legacy Factor - As companies grow, they start building new systems on new architectures and data models. Many of these data structure and models don’t synch up with the old data lying in legacy/core systems.
- The legacy system use data in a way it should not be used, the impact of which is felt, when creators and users leave the organization/function and whole lot of data becomes invisible OR incomprehensible. Apart from day to day work, it makes any system transition difficult.
TIP- Shrink-wrap your legacy applications, and build changes and enhancements in the next generation applications.
Process Dynamics factor: - An organization undergoes business re-engineering. As a result, the existing work-steps are demolished, new work steps and measures are introduced. As the work flow is changes, the additional information is needed and the new stand alone applications are created to fast track the change management. This makes your new processes out of synch with your application systems. By the time these changes are made in your application systems, you will have to manage the data which has been created in your work-around systems. One of the way to manage this data quality root-cause is to include IT representative, as you plan for business re-engineering initiatives.
Time Decay Data Quality issues
- Personal Information: People move addresses OR they change their marital status.
- People don’t fill-up the form and inform every time.
- Value of the assets carried on the books changes in real life, , but not updated.
TIP- As for date instead of years to avoid time decay. Use data augmentation tricks to update the customer records given your time and effort constraints.
Lack of an 'Idiot Proof' system - it is impossible to make any system track all the data quality issues. The systems will slow down, if they do all possible validations and they cannot guard against the deliberate wrong data (which otherwise meet all the domain and standards validations).
Lack of common Data standards/Meta Data
- Very few organizations have the standard methods to record an address OR name OR have the defined domains of the list of cities. There is no common data dictionary.
TIP- Organizations are scared and skeptical of build a single repository of metadata, due to some valid reasons. the more practical approach is to have metadata repositories (including the standards), and integrate them. Refer Metadata Architecture- Detailed scenarios
Deliberate Data Quality Mistakes by the providers of Information
- Customers not giving the right information to safeguard their privacy.
- People giving wrong information to get undue services.
- Lack of stake for the providers to give right OR complete information. (Free mail registration, loyalty program cards etc.)
Data Entry issues - Careless mistakes by the Data entry operator.
- Wrongly designed data entry forms, allowing illegibility.
- System fields are designed to allow free forms OR are not in line with the data requirements (i.e., address field not having adequate length).
TIP- Try to put as many Input Controls as possible
Lack of business data ownership for Data Quality
- There is lack of business ownership of the entire enterprise data OR its defined sections. Very few organizations have the role of head of Information/Data. For Bipminstitute.com, this is the most significant root-cause for data quality issues (and a solution as well)
TIP- Create a business role of data steward and make it empowered and accountable. |
