Challenges of Big Data

Today, all most all marketing magazines, journals, seminars, blogs etc are talking about the benefits of big data. While Big Data promises the gold for a marketer - in terms of having real time customer insights and the ability to influence customer's decision, it is not easy. There are major challenges in implementing really good technology platform needed for Big data analytics.

Getting the vital customer insights is more like gold mining - you need to process 3-4 tons of ore and dirt to extract 5 grams of gold. Big Data analytics will also have to run through tones of data first - except that with advancing technology, it is possible to get that insights fast and cheaper than ever before. Companies must work out the major challenges of Big data first - before they can make use of the customer insights.

Challenge-1: Get the right data source

Today, in most organizations, data is distributed in multiple databases. It is common to see 50-60 different databases. In larger companies, it is common to see data scattered across 100+ databases. So identifying the right data sources for the big data anlytics is the first big challenge. Given the dynamic nature of business today - especially in high-tech world, with mergers and acquisitions, the number of data sources just keeps increasing.

Integrating multiple sources of data is still a big technical challenge and is yet to be solved by IT.

Challenge-2: Choose the right technology.

Another challenge is to select the right technology platform. Today there are several big data technologies in the market: Hadoop, Mongo, MIKE 2.0, NoSQL etc. These new technologies offer an opportunity to analyze very large amounts of unstructured data at a scale and speed that was simply not possible just a few years ago - but selecting the right technology platform is based on the type of analytics you need - which depends on the data sources (see challenge-1)

Challenge-3: Get the right storage IT infrastructure

Big Data analytics needs big data storage and networks to move the big data around. Since the data exists in multiple databases, consolidating data will require a much bigger storage capability and a faster network to move the big volumes of data around the corporate network. In many cases, the current IT infrastructure is inadequate to meet the demands of big data. Data storage & network bandwidth management is still a big challenge for most companies and this has to be resolved before companies can benefit from big data analytics.

Challenge-4: Have an action plan

Big data analytics is useful only if one can act on the analytics. Since most of the real time analytics need real time action, there must be a plan on what to do with the analytics. Today, Big Data is more like a technology hype than a complete business solution with a clear path to business value. So before companies can start on their big data project,  they have to separate the hype from reality and have a clear cut plan on what analytics to run and what to do with it.
Today, most companies don't even act on the customer feedback that is readily available. Marketers don't look at basic Web analytics to act on the insights that have been at their fingertips for years. In such case, should the company be willing to act on the insights you could generate from a Big Data initiative! And for that there must have a plan.

Challenge-5: Avoid the data trap!

Data analytics is not everything. Customers have emotional reactions too and in many cases, customers do not really know what they really need. Henry Ford once said "If I had asked people what they wanted, they would have said faster horses." Customer insights and marketing plans must not rely solely on analytics!

Closing Thoughts

Big Data analytics is a key resource for any marketer and can be a competitive advantage. There are several success stories of businesses using big data to improve the business. However, most companies will hit road blocks on the journey to this land of gold. Today, Big Data is more a piece of the technology than a complete solution with a clear path to business value. Only big technology companies have been able to exploit the benefits of Big Data.

Customer behavior is complex and customers are emotional. So no matter how robust the analytics  is, it can provide only a part of the picture. It takes human intuition - tuned by big data analytics to really make a difference and create a success story.