| You will also see this subject being tracked in the latest trends of bipminstitute.com, and some field-tips on how you should approach the metadata implementations. This page provides different options related to Metadata architecture and commentary on their context.
Some clarifying terms on Metadata Architecture
Is metadata ‘All OR nothing’ enterprise initiative?
Knowing the objective of the metadata, it can be interpreted that a metadata has to have holistic and all-encompassing details on the organization. If you have a partial metadata, it would be detrimental, as half of the organization will follow it and other half will not. For example, if your metadata has the details on the CRM system, but does not have it on ERP system, an IT person has to refer to multiple references on what is going on in these systems. This will reduce the reliability and usage of metadata leading to its slow death.
The fact is that organizations over time have failed to make a truly enterprise wide metadata, due to multiple reasons:
- Lack of metadata standards
- Too big an undertaking in terms of band-width and investment required. By the time you have an enterprise metadata repository, many things and CEOs would have changed.
- Lack of good metadata management tools.
The typical approach for building metadata repository is to create community OR usage based metadata repositories. This ensures that we are able to have more manageable bites.
Metadata communities
Metadata communities OR interest groups are the focused usage groups for metadata. These interest groups create their own metadata repository which will serve their specific interest and will carry details relevant to the group. Following are the examples of the communities:
- BI- Data Warehouse, OLAP, BI end-user tools like enterprise reporting.
- SOA- Service oriented architecture. Contains all metadata related to service based programs, parameters, program interfaces etc.
- Transactional systems: Carries the information on all transactional systems- mainly technical metadata.
- Master Data- Carries the details, standards etc. for master data (both business and technical metadata)
Single vs. Multiple repositories
As mentioned above, a single metadata repository is something which experientially has not been found feasible. In a typically ‘evolved’ organization environment you will find multiple metadata repositories, serving their own interest groups. For example you can have:
- Data warehouse metadata repository (having details around ETL, Models, Data Warehouse jobs, access and security etc..)
- Master-data repository- Carrying the details on the customer, vendor, employee, locations, products..
- Application systems metadata repository- Carries Technical Metadata details on IT applications.
Operational (runtime) vs. Non-operational Metadata Repositories
- Non-operational purpose Metadata Repositories- To support non-operational activities like system design, application architecture, business rules development, data-model development etc.
- Operational Metadata repositories- To support run-time activities. For example, a data warehouse operation checks the metadata repository to find the location of extracted files before it starts doing transformation.
A metadata repository can be designed to provide both the functions. However, in that case the kind of data you store in your metadata repository will be more comprehensive and also your metadata management models (the way you store data in metadata repository) will also undergo a change. For example- for a runtime metadata repository, one will need to store the details on the cache location of a table (as run time program could be faster using a cache). While a metadata repository can support a runtime, it is advised to maintain their focus on non-operational activities.
Metadata repositories vs. runtime registries
There has been some confusion around the runtime registries vs. the metadata repositories. Runtime registries are the datasets containing information for the systems to manage their operations. For example, they contain the details on the cache, active table partitions, temporary and permanent table locations etc. Some of the data in the registries is also found in metadata as well.
- Runtime registries are not concerned about the higher level details like contextual, conceptual and logical levels. They are primarily having physical level details.
- Runtime registries contain additional data which is used purely for running the operations, which a metadata repository does not need. For example cache and runtime log locations.
Metadata Standards
One of the big barriers for an enterprise metadata repository creation in the past is a lack of metadata standards. Metadata standards are having two components:
- Metadata models: the standards metadata storage structure across metadata repositories
- Metadata exchange (OR interchange) standards: the language in which different repositories exchange data.
With advent of XML and SOA, the metadata exchange standards are now reaching a point of robust usage. However, a common set of metadata management models is still developing.
The Architecture Options for Metadata Repositories
Hub and Spoke Repositories
Given that a single metadata repository is a big-bang initiative infeasible for an organization, the organizations have moved to hub and spoke architecture.
- Hub: This is a central point which is a router for someone to access a repository. It can also contain the standards related to the management of the repositories like XML schemas and shared metadata.
- Spoke: this is the actual metadata repository serving a given interest group (like data warehouse metadata repository). It interacts with the other repository through the Hub. For example, if you are a Data Warehouse designer and you want to check on the business rules related to a master table in one of the source system. In that case you may do to the metadata repository of Master Data to get that detail.
Federated Metadata Repositories vs. Metadata Repository warehouse
Moving on from Hub and Spoke model, one can have following options in this model:
- Virtual Hub: A hub containing the standards, XML schemas and taxonomies to help spokes talk to each other and work more consistently. It will not contain any actual metadata.
- Shared metadata hub: This is virtual Hub + shared metadata, which is most extensively used.
- Metadata repository warehouse: This contains the actual metadata picked and integrated from the spokes. It is similar to a data warehouse where that source systems are the spoke metadata repositories.
|
