- Intelligent Data Health Assessment
Cognitive technology for the DataHub to identify data health issues and "self heal"
Over the years implementing 3NF and Dimensional Data models for Banks, we came to realize that "data quality" is subjective based on the institution and the end-use of the metric/attribute. Traditional ETL design will trigger an alert for every failure and waste organizational effort in the process.
We set-out to develop a data quality assessment engine that is intelligent enough to answer the question – "Ok, something's not right. But, can this wait till tomorrow morning?" The engine should be train-able to understand the importance of a report and the importance of the dimension for that report.
The ETL "self-heal" has saved several 1000 person hours of ETL operations time and we are aiming higher.