Big Data project and challenges

Even before you build the unified data warehouse for your company, my suggestion is to establish a framework to measure the quality of data that is there in the on-premise systems before even it flows to big data systems. Many applications both internal and external to the companies provide data in various formats and they must be finally transformed to a standard format to measure the quality of the data starting from veracity to value of the data. Sometimes the applications have some bugs and they logged inconsistent data in the past and now it may be fixed. We need to account for variances like these. Overall it is very important to measure the quality of your data using the 5V’s from Gartner’s terms.

1. Volume
2. Veracity
3. Velocity
4. Variability
5. Value

Data cataloging and metadata about the data is very very important if we want more people in the enterprise to use the data. I can give you few examples from web services and API world. When web services were developed they had directories called UDDI – Universal Description, Discovery, and Integration. This had the list of all the web services that can be consumed by applications and it helped developers find relevant methods to use for their requirements. Similarly each big data system offers its own data catalog features and allows developers to enter metadata, tags. This must be followed deligently and it will help users find the right and relevant data need for them to solve the problem at hand. There should also be a way to let users know what data is missing in the system that they will need it in future.

There are lot of privacy compliances that companies need to follow to protect the personal, payment and other information about users. Anonymizing the data of the users at scale is a challenge. Sometimes the data is anonymized in different ways by different applications and this poses a challenge as it makes it unusable when it is merged in the unified data warehouse with other data. Companies should think of having a standardized encryption, decryption standards and anonymization routines. This will make the data more usable in anonymized formats.

Data science teams always ask big data team for best slice of data to train their algorithms. Sometimes they ask for both training set and testing set of data. Many times big data team will not be able to slice the best possible data for the training because of inconsistency in the data and also team members may not be aware of what tables and what columns must be used in the extracted data. This is a challenge that creates friction between big data and data science teams and companies should make both the team work in harmony by implementing catalog standards, documentations, queries etc.

It is very tough to trace the origin, flow and destination of the records. Traceability is impossible in the big data world. We cant debug the system that easily. Have a traceability of data outside of big data system and ensure that big data is receiving data correctly. Be proactive in inserting/updating data in the big data systems.

Before the users start using the data from big data system we should verify that the numbers in reports are same between the old system and new big data systems. I see that companies that skipped phase of verification end up losing trust of the users. If the users somehow find out that numbers are not lining up then it will create a cascade of dominoes and users end up losing trust and then eventually the system becomes useless. Mostly when the data science team could not increase their accuracy with this data then it will prompt users to lose trust in the data.

When initiating projects to create unified data warehouse please keep in mind the challenges listed in this post and plan accordingly.