6 stages of data transformation

We have worked on projects that were top-down driven strategic projects to help clients in their digital transformation initiative. One of the key things we have noted for successful execution of the projects is that their team has the mindset that everything is a data for them. To them the data is a raw diamond which needs to be extracted, cut, polished and made it more useful or like gold where it is blended with other metals to make it more useful for everyday use. This is the first topic we train our clients and prospects whenever we undertake any big data and analytics project. 

The first strategic project in data-driven digital transformation is building a unified data warehouse. Today the organization structure is not hierarchical with huge silos between various functional units. In some of the projects that we worked on stakeholders from various functional units would attend daily standup for eg: Programmers, Business Analyst, Customer Support, Presales, Marketing, CFO, Human Resource department people. In order to become more agile companies are now blending resources across their organization to create the best teams to deliver the outcome. The diverse set of stakeholders need to see the data brought out of its silos into a unified central place. This is a must. In some companies that we worked with they had data in various systems like Marketing and Sales had their data in CRM, Finance in Tally or Sage Accounting software, IT had their data in databases, Business users had their data in the Excel and other file formats. Externals vendors were uploading the files in plain text, JSON, AVRO and other formats. To empower users to see the big picture and impact across all layers of their organization both vertically and horizontally they needed data from everywhere in the company. We help clients build this unified data warehouse as first phase of the development project for big data and analytics.

Once the unified data warehouse is created then we moved on to migrate the reports that run off of the silos data sources to the unified data warehouse. We mostly host the unified data warehouse in Google BigQuery, Amazon RedShift and Azure Synapse Analytics and we are exploring few other systems for very unique regulatory, privacy requirements. This frees up lot of enterprise resources that were previously used and the enterprise was able to allocate those resources for transaction processing and other run other processes. Since the unified data warehouse is now in cloud multiple reporting tools like SSRS, Tableau, Power BI, Google Data Studio, Apache Superset and others can be easily integrated to empower users to design and develop reports. Mostly in this phase we migrate only the predefined reports that were already running. This will help us guage the billing in cloud for the unified data warehouse and also help IT team put some restrictions and quota usage to control the billing and keep it predictable every month.

This is where everyone gets a pass to access the data that they need for the problem they are trying to solve.. Users start to use the data everyday. They look at what happened at a day level, week level, month level and compare year-over-year details. We also have enough privacy and security measures in this phase. Usually at this stage we do a facilitation workshop rather than training to help users create the reports and alerts that they want for their use. At this point data is used strategically through out the organization. There are still few challenges users will be facing most of which fall under the category of how to find the data relevant for their problem.

One of the criteria of a profitable customer in retail is to create a self-serving customer. If you look at Amazon it is a completely end-to-end self-service where customer order the product and get updated from the app and then even initiate return and refund without every talking to the customer support. Similarly insurance policy purchase are now completely self-service where user takes picture of their cars and then a computer vision system detects damages that are already there and then confirms it with user and gets complete history of the driver and vehicle from previous insurance and the calculates the premium and helps customer pay for it without ever talking to a representative. We build a data platform at this stage of the data transformation where users can find out the relevant data they need for their problem at hand to solve and the use it and apply various what-if scenarios on that and help them not just use data to report about the past but about why the past was that way. This is the phase were customer start to get their return on invesment for the data-driven digital transformation project. Users are empowered with the data and tools and they free up the tech team’s time and do tasks indepdendently. 

Now that users are using the data on a day-to-day basis and a thriving data-driven culture is created the users become so smart that they can easily figure the missing piece in the puzzle. They know what data they need from outside the company for them to blend it with their data sources and use it to make effective decisions that are free from bias and that does not have any blindspots. At this stage we encourage clients to go out and buy the data they need and integrate it with their data sources. This is a very valuable phase for introducing artificial intelligence in their company. The models that the machine learning algorithms trains needs data more wholistically and hence this is important.  For the sustainable goals that companies are setting every year they need data about climate change, environment and other natural resources. These data cannot be collected by every company so we recommend buy it from a provider. 

Knowledge in the past was measured about how well someone know the past and bring it back to memory and apply it. Today that has changed. Knowledge is now measured as how well can someone predict a future. This is where companies start to use the data from past to predict the future or identify the variables that play a key role in determining the outcome of a problem in future. Predictive analytics transforms the companies to be proactive. Take small problems and use predictive analytics to predict and improve accuracy. A feedback loop must be established to train the machine learning algorithm and improve the learning and predict more accurately. This is the phase where companies start to explore cognitive automation and use of artificial intelligence for their company.

Hope this post provide an overview of the journey to build data-drive culture in a company. This 6 stage transformation does not happen overnight. People, process, system, techonology all have to be weaved together with support from top management to drive this data-driven digital transformation. If you need help with data-driven digital tranformation initiatives in your company then you can reach out to us.