Data lake - Solving the challenge of Big Data Integration

Big Data analytics is a game changer for businesses today. Unfortunately, most organizations are struggling with collect & integrate vast volumes of data needed for business analysis. As a result, with poorly integrated sets of data undermines business analysis and executive decision making process.

As organization start to implement their big data analytics projects, the first step is to develop a comprehensive strategy for managing data:

• A strategy that should incorporate all sources of data needed for analysis. 
• A strategy that should incorporate capable technology & tools for big data
• A strategy that make data integration in a smooth & fast to provide timely analysis.

Companies that are well equipped for big data integration will operate more efficiently and effectively. Data lakes enables companies with new generation of technologies - which is the first essential step to increasing agility in business.

The challenge

Organizations are seeing huge increase in volumes of data. Data is coming from various sources:

1. Structured data from databases, web pages, OLTP, etc.
2. Employee created unstructured data in form of files, emails, IMs, etc
3. Machine generated data from sensors
4. Video surveillance feeds
5. Misc. user generated data: Photos, videos, pdfs etc
6. Data from external feeds: Social networks, Twitter, news sites, web comments etc

As types & sources of data increases, the challenge of data integration multiplies. The traditional data warehouses cannot cope with new types of data and is not designed to handle this high volume and variety of data. As a result, the traditional BI tools fail to give meaningful insights for decision making.

In the world of big data, Legacy BI tools are slow and error prone.  There is a widespread dissatisfaction with their current data integration technologies and organizations are finding it too slow and hard to maintain data.

According to a study done by Ventana research:

• 78% of organizations are facing challenges in integrating different data sources.
• 55% of companies are somewhat confident or not at all confident in their ability to process lage volumes of data
• 58% doubt their ability to process data that arrives at high velocity. 

Organizations waste significant amounts of time on data integration tasks, particularly in reviewing data for quality and consistency, which is needed to prepare it for business analysis.

Data integration must be fast and accurate for market place agility. Most organization need data on hourly or daily basis. In Internet economy, real time data analytics is the key for success.

It is critical that data integration and data ingestion capability to be flexible enough to deliver multi cycles of processing to satisfy different analytical needs - i.e., to be used by wider big data analysis.

Use of public cloud for applications also complicates data integration. Organizations are having a mix of public cloud and on premise IT - which essentially complicates data integration and timeliness of data for analysis.  Accessing data in traditional batch cycles is not the best way to utilize cloud data sources.

As a result, companies are looking for tools to automate data integration.

EMC Data Lake Foundation

EMC Data Lake Foundation with Pivotal Suite of Big Data analytics can address most of the data integration challenges.

EMC Data Lake Foundation - which is based on EMC Isilon and EMC ECS (Elastic Cloud Storage),  integrated with the rich analytics tools from Pivotal can provide a common integrated data pool  - thus make it simple to collect, store and  analyze massive volumes of data.

EMC Data Lake Foundation solves the problem of data integration by providing a common data lake that accommodates both high velocity unstructured data, machine data and tradition databases. With Pivotal suite of analytic tools, while leveraging existing BI tools in the mix allowing existing business analytics to work along with new Big Data analytics.

Unified Data Lake is the game Changer

Creating a unified Data Lake which had ingest and hold both traditional data in existing data warehouses and newer data types should be the first step while embarking on a big data journey.

A unified Data Lake gives companies a choice of data extraction and analytics tools and does not lock workers into using old existing solutions, Older workflows can be easily integrated with newer Big Data analytics workflow.

A unified Data Lake allows new Big Data analytics solution can use  new technologies like Hbase, Storm, Hive, Pig, Mapreduce, Gemfire, etc to provide analytics for different applications, while providing enterprise class data security, protection and access control in a centralized, integrated way that data is accessible and easily managed.

A Unified Data Lake offers several benefits including:

• Agility: Eliminating much of the strain on IT that was common with traditional silo approaches.
• Simplicity: Allow consumption of data in any format, thus saving time & reducing errors
• Flexibility: Allow for the use of different analytics techniques, mix of both old and new, which helps organizations see the data differently to ask new questions and derive new insights
• Accessibility: Provide users with fast, easy and secure access all their data.

Closing Thoughts

A well thought out data integration strategy with EMC Data Lake foundation will enable companies to reap the full benefits of Big data. The data lake allows companies to:

1. Retain and analyze more data
2. Increase the speed of analysis
3. Secure business data with enterprise class data security systems
4. Meet business needs for decision making
5. Make more information available across organization

Integrated data lake will maximize the return on investments in big data analytics.

Minimizing Risks in New Product Development

All too often, we hear about risks and failures with new product development. Developing new products is a complex, expensive process and has unpredictable outcomes. In other words, new product development is fraught with risk.

Every week, thousands of new mobile apps are being developed. But a handful of them make any money. If you are developing a new product - even with a disciplined business plan, a well managed project management and all other business processes in place, new products can still have unpredictable results. Even successful/innovative companies have seen product failures.

It is almost impossible to eliminate the risks associated with new products, there are few things one must do to minimize the risks. It begins with aligning the business goals with a proven technology possibilities to produce successful & innovative products.

1. Develop a pragmatic business goals.  

Apple sold 26 million pieces of iPhone 6.  So, if you are a competitor to iPhone, and define your business plan is to sell 20 Million phones a quarter, then the plan is set up for failure. Instead develop a pragmatic plan which takes into account of various factors: Marketing & Sales channels, supply chain, etc.

The plan must be something achievable, realistic, and consider the inputs from other parts of organization.

Start by clearly defining and articulating the business plan and validate it through objective analysis before embarking on significant new product development activities. The business objectives should be unemotional, objective and quantifiable. This helps us in a big way in decision making during the project and not waste time.

2. Don't Rely Exclusively on the Voice of the Customer.

Contrary to existing conventional wisdom, relying exclusively on the voice of the customer will often steer you on wrong direction. Customer often adds his wish list into the requirements - just to see if it can be done. Customers may not always know what they really want, and they rarely know what is technically possible.

Instead of listening solely to what customers are saying, also look at what customers are willing to pay for. Identify what is the bare minimum features that customers will pay for and then use your instincts, intuition & market knowledge to identify all the required features.

This makes it possible to reliably develop the new products in cost-effective way and that can be sold to customers.

3. Bring proven technologies from other companies. 

Most new products needs new technologies. Current players may not have developed the product idea you have - because the current technology/platform may not be capable. So the first step is to evaluate new technologies and adapt proven technologies - mainly from other companies. Typically, startups often test new technologies and it makes sense to copy it fast  if the technology is already proven.

Large technology companies acquire startups for their proven technology. For example, Cisco acquired Meraki & EMC acquired iWave etc.

Applying proven technology from other company reduces the risks while enabling rapid product development.

4. New Product Development requires disciplined project management.

New product development requires innovation. And innovation typically has a reputation for uncertainty and unpredictability. By approaching innovation with a pragmatic and disciplined project management process, it takes away the risks. If the innovative project does not deliver expected results or fails to meet the milestones, then having a disciplined approach helps to identify risks ahead of time and take corrective measures - even if it implies canceling the project.

A disciplined project management implies:

• Knowing the key milestones and final objectives
• Project plan calls for measuring the pace of innovation
• Align innovation with new product development plans
• Keep the project on course when distractions occur.

The discipline to spend time up front helps reduce the risks associated with the creative and technological sides of innovation. This will ensure that what goes into the project is truly meaningful and will yield the desired business results at the end, when the new product arrives on the store shelves.

Project management is an upfront investment to avoid "garbage in, garbage out". Defining all the inputs and expected outputs in the project will avoid wasting resources and minimizes risks.

5. Create an agile development team 

New product development requires an ability to change if the market needs change. Ability to read the market needs quickly and adapt to the changes is critical for success. Agility starts at business leadership level and must spread across the organization, including marketing and technical teams.

One must constantly be monitor what's going on in the world, what are the trends in technology and marketing? And ask "How does this impact our product?" "How should we adapt to succeed in this?"

New product development requires agility from everyone involved.  

Data-Analytic Thinking for Leadership

Today it has become important for all leaders to understand data science and impacts of data science - even if they would not develop data analytics application. Leaders need to think and make decisions. So data-analytic thinking helps to take better decisions. Even politicians, CEOs and NGOs are using data analytics in several ways to make decisions. The potential for success is greatly enhanced with data driven decision making Vs instincts. Even instincts can be validated with data.

Today, there is plethora of data sources and data is everywhere, information is now widely available on web pages, news channels, social media, events data and internal data in CRM, ERP,  Emails, etc. All this broad sources of data has led to increasing interest in methods for extracting useful information and knowledge from data.

Virtually every aspect of business & life is now open to data collection: Operations, Manufacturing, stock market news/trends, supply-chain activities, social media, new channels, sensor data, mobile phones, traditional IT systems, and so on.

Getting all the required data to access a problem needs careful planning, systematic data collection and analysis - and is called as Data Science.

Ubiquity of Big Data Opportunities

Data is everywhere and it comes from variety of sources and different speeds and sizes. Managers, leaders in almost every industry are focused on exploiting data for competitive advantage. Historically, this was done by a dedicated team of Business Analysts - who would use teams of statistics, data modeling tools to explore & analyze data manually. But with the digital world, the volume & variety of data have far outstripped the ability of manual analysis.

At the same time, computers have become more powerful & cheaper. Tools have been developed to collect data, analyze data to enable broader and deeper analyses than previously possible. The convergence of the ability to compute & big data has let to widespread use of data science & data mining techniques.

Probably the most well known applications of data-mining techniques are in marketing for tasks such as targeted marketing in online advertising, and recommendations for cross-selling in online retail.

Big data Analytics is also used for innovation and scientific discovery. The ability to rapidly analyze thousands of genomes is leading to new science of pro-biotics, where bacteria is being used to cure diseases!

Within an organization, virtually every department can use big data analytics: Analyze customer behavior, predict employee attrition, minimize customer churn, predict future interest rates, predict future supply-chain demands, High frequency algorithmic trading etc.

In short, companies are using data analytics as a competitive differentiator. As a result, a new business function is being formed within all companies: Data Analytics Group.

The primary goal of data analytics group is to help leaders view business problems from a data perspective and bring in data-analytic thinking as an input to decision making process - which is in addition to: intuition, creativity, common sense, and domain knowledge.

A successful leader will use data analytic thinking in decision making tool kit. 

Value of Social Media Analytics

This weekend, a newly hired marketing manager asked my opinion on social media analytics. His company is medium sized construction company and had recently ventured into housing business. As I spoke to him and advised on social media analytics strategy, I thought of documenting the gist of what I said in this blog.

Today, every company in fortune-500 list is using an array of social media analytics - from a host of data analytics services companies. Even smaller companies with revenues in less than $10M are now buying social media analytics.

Social media companies are also making it easier for companies to collect data through APIs & its free. Facebook insights, twitter analytics, Google analytics etc. are so easy to use - that even small start ups can benefit from it.

So this raises a question: Is the custom built sophisticated big data analytics any better than the free versions from Facebook, Twitter & other social media?

At the first level, paid analytics looks more sophisticated and definitely looks good. Vendors offer pretty reports and better interfaces. For a higher price, few companies offer better math that is used to quantify ambiguous metrics - such as depth of customer engagement or Potential reach of the product.

A common justification for paying for social media analytics is usually:

 "Customers are very important to us, so why not spend some percentage of marketing spend to know & understand what customers are saying."

"Hearing what customers are talking - will help us better position out advertisements. Even a 1% increase in customer response to an advertisement leads to 10-15% increase in sales." 
 
While there is no denying that there is value in social media analytics, businesses must spend considerable time to conduct rigorous data modeling ahead of time to ensure that the metrics provided by a third-party analytics program/services are meaningful to the business. Without an upfront data modeling - you will end up with a flood of data and reports - that does not tell anything new.

Data modeling starts with identifying the right data - Not just data. You need to filter out spambots, auto retweets and clickbots etc.. Next, you need to identify keywords or phrases that are relevant to your business. For example word "suck" is generally bad for retail, but a good word for vacuum cleaners. It takes time to identify which phrases are "positive," "neutral" or "negative".

Keyword based analytics can give huge value - but only when it is done right. Businesses need to spend time/effort to identify keywords & refine keywords on periodic basis.

Once data modeling is done, the next is to firm up on the strategy:

• How to deal with the information you are getting from social media feeds?
• How to manage customer complaints?
• How to increase traffic with social media?

In the world of social media, the time to respond to customer complaints and negative comments is really short. So businesses needs to have a response strategy in place - before starting off on social media analytics.

Closing Thoughts 

Unless businesses have done their homework in terms of data models, response strategy, It is best to use the free basic metrics provided by the social media platforms. Only when you have a strategy - then invest in social media analytics.

Product Leadership - Building Successful & Innovative Products

One can learn good management theory and principles in MBA colleges. One can learn creativity and design in design institutes. But no one teaches how to create successful products!

Creating successful new products requires a right marriage of right brain thinking:  creativity/design, and left brain thinking: Process, science/maths, rational management. Apple Inc. is a perfect example of left brain right brain thinking - Jonathan Ive the designer and Tim Cook, the operations manager. Steve Jobs the business leader.

Every company that develops new products needs product leaders managers who can get the right balance between creative people and managers to create successful products.

Product leader has to bring in few tricks: "Design thinking" and "Lean Development"  techniques to rapidly experiment with solutions.

The secret for success in new product development is to understand that developing new products is not much about product development! Instead, it is all about creating customers!

Creating customers is all about understanding customer needs, desires, & problems, and solving it profitably. If the number of customers are large enough, then you have created a successful product.

Since there is a very high level of uncertainty in creating new customers, one needs to be agile & lean when it comes to trying out new designs - i.e, ability to move/change/adopt quickly. Since things are at flux and needs are constantly changing one has to be lean - i.e., use minimal resources for experimentation.

Startups generally are good at new product development - mainly because they are not focusing on sustaining customers. They are creating new customers. They have few resources & are forced to be lean and they are agile. Enabled with rapid decision making leadership, who is not afraid to experiment or fail, Startups are constantly figuring out how to create a customer.

I always say "Without a customer, you don't have a product. You only have a prototype!"

I have worked in startups and have seem several other start ups and established companies follow a similar process towards new product development. Often

Step 1: Develop an Insight: 
Identify what the market needs or desires. Start with talking to customers, question customers current levels of comfort & needs. Observe customers and learn. Network with similar minded people. Also be ready for surprises. Lastly experiment - to get deeper insights about problems that is worth solving.

Step 2: Define the Problem: 
To develop a successful product, one needs to know what is the real problem that needs to be solved. Competition could also have the same insight - but had failed, because they did not solve the right problem.  Discover the customers' need or problem first, and then make sure you are going after a problem worth solving.

Step 3: Solution: 
Develop a prototype first. Check if the prototype meets the minimum requirement in terms of solving the customer problem. Practice agile & lean principles to rapidly develop prototypes - instead of developing full scale products. If needed, develop multiple prototypes, each to solve many aspects of customer problem. Iterate until you develop an awesome product - the one that delights customers.

Step 4: Develop Business Model: 
The finally solution must be priced to market. Carefully examine the entire cost structures, validate go-to-market strategy. If the need be, one has be creative in pricing the product. A differentiated pricing strategy that makes it easier for customer to acquire the product.

Closing Thoughts

Product Leaders must push the design and management teams to go beyond the obvious. Force the team to Think differently or experiment. Accept failures during prototype stage - and fail fast. Once a failure is identified, pull out the product and iterate until the product succeeds.

Leaders must embrace uncertainty in the product development, Challenge the problem statement until every stakeholder is convinced of the problem, Set a high pace for development, and be bold in experimentation.