Part 1: Big Data, Big Misunderstanding
Big data is seriously misunderstood.
As a consultant in business intelligence, I find myself constantly explaining the concept of big data to my clients. It seems that regardless of their background, I’m frequently met with unfamiliarity, or significant misunderstanding of what big data is, and how it’s affecting the competitive landscape of nearly every industry. Being unfamiliar with big data is dangerous. Being misinformed is even more dangerous.
This blog is designed to dispel the confusion around big data, and to answer the questions critical to IT departments, business executives, and the front-end users of data.
Big data is the most important and misunderstood advancement in the short history of business intelligence.
Let me step back from the oft-repeated “Three V’s” of big data: volume, variety, and velocity; the technical issues of maintaining large data stores; and the real-time requirement of today’s analytics; all issues of which will be discussed in-depth in this blog. Before I dig into that, I first want to explain the benefits of big data as lucidly as I possibly can.
Big data allows companies to access massive volumes of data to identify important trends and patterns. This uncovers causations and correlations between previously unconsidered metrics, without needing to know the right questions to ask. Get it? Good.
In the previous world of structured, centralized data, one had to have straightforward questions in hand in order to obtain meaningful information. Structured data is great for questions such as, “What was last year’s margin, by product, or by region?” Big data and predictive analytics allow us to ask questions such as, “What are my customers looking for when visiting my website?” or better yet, “What products should I suggest to a particular customer when they log onto my website?”
Amazon knows I have a dog, and I’m sure I have never directly told them this fact about my personal life. They derived this, not only from my purchase history, but also from the products I’ve browsed and did not select. Think of the competitive advantage this gives Amazon over a brick-and-mortar store. Amazon uses an algorithm to suggest not only the products I’m most likely to purchase, but also the products that return the highest margin to Amazon. This is the proper use of buyer-specific big data.
The ability to properly utilize multi-structured data created from social media, server logs, images, and equipment sensors, to name a few, will allow organizations to anticipate an issue before it upsets operations or sales, instead of explaining to shareholders why the company missed projections.
Twenty years ago, Lew Platt, former CEO of Hewlett Packard, said, “If only HP knew what HP knows, we’d be three times more productive.” If your company is still debating, or is yet to implement, the use of advanced analytics, you are at least twenty years behind Mr. Platt and the wave of original thought in knowledge management, business intelligence, and predictive analytics.
The first step to achieving data competency in the age of big data is recognizing that your organization is data deficient. As of today, you do not even “know” the data that you currently store, and have no idea where to begin to tackle the volume, velocity, and variety in the wave of big data coming your way. This wave is massive. In fact, according to IBM, 2.5 quintillion bytes of data are being produced every day, which means 90% of the data in the world has been created in the past two years alone. Furthermore, Forrester Research estimates that organizations effectively utilize less than 5% of their data. This means that 95% of what your organization “knows” isn’t being used to its full potential.
In my next post, we will discuss the problem of data deficiency, data “democratization,” and the veracity of insights obtained through big data and multi-structured sources of information.
Big data is a significant change agent in your marketplace, regardless of industry. So, until our next post, remember, “When you’re finished changing, you’re finished.”
Dan Grace is a Consultant II at SDLC Partners, a leading provider of business and technology solutions. Please feel free to contact Dan at firstname.lastname@example.org with any questions on this blog post or to further discuss Big Data.