The War for the 4 Pillars:
Big Data, Cognitive, Cloud & IoT

Companies are dealing with an explosion of data and are trying to make sense of all of this. And software super vendors like IBM are jumping into the fight by providing everything needed to implement these 4 pillars. They provide whatever you need to host (in the cloud), capture, manage and analyze your data that will hopefully ease decision-making, facilitate efficiencies and provide a competitive advantage in this global marketplace. Amazon, Microsoft, IBM, Oracle and SAP are some of the other super vendors that are connecting these dots. I believe there will be lots of consolidation of software vendors. Acquisitions of smaller vendors with complementary technologies gives each super vendor a temporary competitive advantage and is the only way for the smaller vendors to avoid becoming road kill over the long haul. Smaller vendors with competitive software are in for a rough ride.

The global managed service providers are adjusting strategies to prepare for this war too. Led by companies like IBM Global Services, Wipro, Accenture, TCS, Capgemini, Infosys and others, they will be assisting their clients with implementing solutions to fit these needs. Whether it be moving data from mainframes and flat files to big data lakes and/or data warehouses, or migrating from one big data vendor to another, there will be from terabytes to petabytes of data involved.

All of this assumes the underlying data is in reasonable shape. But is it really? What of data governance and data quality? Why is that lagging behind?

It reminds me of the early days of automated software testing (see article here) where interesting development processes of the time, like RUP and XP, were moving developers away from the waterfall process. Yet the support for automated testing lagged way behind. And without test automation speeding up the process, it caused critical applications to be partially tested, leaving them extremely buggy — resulting in huge delays in delivery and being way over budget. That seems to be where everyone is now with data quality and governance, but help is on the way. There is an Apache project named Atlas that is focusing on data governance. And there are solutions that automate the validation and testing processes — solutions like QuerySurge and others. But right now, the quality of corporate data is sketchy at best and big data lakes are in danger of becoming swamps.