Data lake - Solving the challenge of Big Data Integration

Big Data analytics is a game changer for businesses today. Unfortunately, most organizations are struggling with collect & integrate vast volumes of data needed for business analysis. As a result, with poorly integrated sets of data undermines business analysis and executive decision making process.

As organization start to implement their big data analytics projects, the first step is to develop a comprehensive strategy for managing data:

• A strategy that should incorporate all sources of data needed for analysis. 
• A strategy that should incorporate capable technology & tools for big data
• A strategy that make data integration in a smooth & fast to provide timely analysis.

Companies that are well equipped for big data integration will operate more efficiently and effectively. Data lakes enables companies with new generation of technologies - which is the first essential step to increasing agility in business.

The challenge

Organizations are seeing huge increase in volumes of data. Data is coming from various sources:

1. Structured data from databases, web pages, OLTP, etc.
2. Employee created unstructured data in form of files, emails, IMs, etc
3. Machine generated data from sensors
4. Video surveillance feeds
5. Misc. user generated data: Photos, videos, pdfs etc
6. Data from external feeds: Social networks, Twitter, news sites, web comments etc

As types & sources of data increases, the challenge of data integration multiplies. The traditional data warehouses cannot cope with new types of data and is not designed to handle this high volume and variety of data. As a result, the traditional BI tools fail to give meaningful insights for decision making.

In the world of big data, Legacy BI tools are slow and error prone.  There is a widespread dissatisfaction with their current data integration technologies and organizations are finding it too slow and hard to maintain data.

According to a study done by Ventana research:

• 78% of organizations are facing challenges in integrating different data sources.
• 55% of companies are somewhat confident or not at all confident in their ability to process lage volumes of data
• 58% doubt their ability to process data that arrives at high velocity. 

Organizations waste significant amounts of time on data integration tasks, particularly in reviewing data for quality and consistency, which is needed to prepare it for business analysis.

Data integration must be fast and accurate for market place agility. Most organization need data on hourly or daily basis. In Internet economy, real time data analytics is the key for success.

It is critical that data integration and data ingestion capability to be flexible enough to deliver multi cycles of processing to satisfy different analytical needs - i.e., to be used by wider big data analysis.

Use of public cloud for applications also complicates data integration. Organizations are having a mix of public cloud and on premise IT - which essentially complicates data integration and timeliness of data for analysis.  Accessing data in traditional batch cycles is not the best way to utilize cloud data sources.

As a result, companies are looking for tools to automate data integration.

EMC Data Lake Foundation

EMC Data Lake Foundation with Pivotal Suite of Big Data analytics can address most of the data integration challenges.

EMC Data Lake Foundation - which is based on EMC Isilon and EMC ECS (Elastic Cloud Storage),  integrated with the rich analytics tools from Pivotal can provide a common integrated data pool  - thus make it simple to collect, store and  analyze massive volumes of data.

EMC Data Lake Foundation solves the problem of data integration by providing a common data lake that accommodates both high velocity unstructured data, machine data and tradition databases. With Pivotal suite of analytic tools, while leveraging existing BI tools in the mix allowing existing business analytics to work along with new Big Data analytics.

Unified Data Lake is the game Changer

Creating a unified Data Lake which had ingest and hold both traditional data in existing data warehouses and newer data types should be the first step while embarking on a big data journey.

A unified Data Lake gives companies a choice of data extraction and analytics tools and does not lock workers into using old existing solutions, Older workflows can be easily integrated with newer Big Data analytics workflow.

A unified Data Lake allows new Big Data analytics solution can use  new technologies like Hbase, Storm, Hive, Pig, Mapreduce, Gemfire, etc to provide analytics for different applications, while providing enterprise class data security, protection and access control in a centralized, integrated way that data is accessible and easily managed.

A Unified Data Lake offers several benefits including:

• Agility: Eliminating much of the strain on IT that was common with traditional silo approaches.
• Simplicity: Allow consumption of data in any format, thus saving time & reducing errors
• Flexibility: Allow for the use of different analytics techniques, mix of both old and new, which helps organizations see the data differently to ask new questions and derive new insights
• Accessibility: Provide users with fast, easy and secure access all their data.

Closing Thoughts

A well thought out data integration strategy with EMC Data Lake foundation will enable companies to reap the full benefits of Big data. The data lake allows companies to:

1. Retain and analyze more data
2. Increase the speed of analysis
3. Secure business data with enterprise class data security systems
4. Meet business needs for decision making
5. Make more information available across organization

Integrated data lake will maximize the return on investments in big data analytics.