Integrated Metadata
Metadata architecture should be well-integrated with the sources as well as the users of the metadata. Any form of integration is complicated. Like ETL in a data warehouse, integration part of the Metadata architecture is most time consuming. Metadata integration can have various forms (please refer to Metadata Architecture Scenarios), like Physical Integration, Federated Integration, Point to Point bridging etc. Integration is enabled by:
Standard Metadata Management Models:
Just like foundation dimensions, one needs to have the standard metadata models on how you are going to store the metadata information. For example- A model to store information on physical tables in your IT applications.
TIP- Define your metadata management models in a very comprehensive way so that any kind of metadata can be absorbed by it. You may like to refer to industry defined metadata models.
Standard Metadata Tool features:
If different metadata repositories, which exist in your organization, have different features and methods, it will be a challenge to integrate. For example- If the semantic layer data structures (the structure in which the semantic layer information is stored) are different, it will be a challenge to integrate. One other example is - Tools handling the versioning in a different manner.
TIP - While you may not be in a position to stop the organic growth of multiple metadata repositories, you can enforce a single tool. The evaluation and selection of the tool should ideally to be managed by the CIO.
TIP - Preferably skew towards the metadata offering of your core enterprise application vendor, as long as it has got decent capabilities and well-defined road-map.
Using established metadata exchange standards:
Many vendors, who state that they have metadata exchange capabilities, work well within their own suite of applications. Integrating with other technologies is still a challenge. XML is coming out as a good choice, but many of the metadata tools are not fully XML ready yet.
TIP- During Vendor Evaluation, Enquire from your vendor on how the metadata tool links up with other technologies and what Meta-data exchange method it uses.
Enterprise Level Readiness Metadata Architecture:
Architecture should be built in a way that it will work for an enterprise level metadata initiative. Most of metadata Initiatives are typically enterprise level, but they could be for a specific purpose like for Data Warehousing. The enterprise level readiness will prompt you to:
- Create more comprehensive metadata models across the enterprise
- Creating greater flexibility for metadata integration from varied sources.
- Your tool selection will be driven by more comprehensive capability expectations.
Scalable and extensible Metadata Architecture:
Metadata architecture should be able to absorb advent of new systems, new BI environments and also new organization environments (like mergers and acquisitions). Scalability and extensibility is linked to:
- Handling increased volume of extraction, transformation and loading jobs.
- Handling large number of connects with diverse metadata sources
- Handling large number of users (physical users as well as the applications)
- Ability handle changes in metadata management models
TIP- Ask your metadata management vendor to provide the benchmarks of performance.
Robust Metadata Architecture
Following are the standard aspects of robustness which will apply to any IT application
- Error handling including roll-back
- Run time Logging
- Back-up, archiving and purging
- Crash and recovery management
- Robust batch job management
- Ability for smart queuing in case of sudden surge in volumes
Open:
This is linked to the integration part as mentioned above. However, openness goes beyond a common metadata exchange standard (like XML) between repositories. The open architecture should be able to have:
- An ability to link with diverse metadata sources and have metadata exchange capabilities with external entities.
- Using open platforms in terms of databases. There are some metadata management tools which have their proprietary database structures.
- Using non-proprietary languages, therefore allowing companies to access wider skill base and easier customization.
Automated:
Metadata has diverse types of Metadata (technical and business). You can manage your metadata repository, only if most of the tasks are automated. However, just like a data warehouse, a lot of up-front effort is needed to
- Define metadata models.
- Specifying the ETL for metadata warehouse OR designing a data federation (virtual and non-physical integration, where data is lying in the independent metadata repositories).
- Metadata operations etc..
Once the up-front design activities are done, the updation of the metadata environment should be mostly automated. Manual updation of the metadata is a very arduous task. It should be generally limited to the contextual OR conceptual level.
TIP- As you design your metadata architecture, identify the areas where it is not possible to automate the updation, and prepare your plans accordingly. Do not assume that you will be able to handle that manual task easily. If the manual updation is critical, get long-term management commitment and establish robust operating mechanisms.
|
