Monday, November 16, 2015

Building World Class Products

When we think of building world class products, few companies come to our mind instantly. The top one for most people is Apple, Microsoft, Google, Coca Cola, Toyota, McDonalds etc. These companies are well known for their successful products - super successful products.

When we look deeper into these successful products, we also find several common traits across these products - which helped them successful in the first place.

In this article, I will talk about one common factor: Product Development Process.

Successful companies have developed a well defined process for developing new products which at a high level can be broken down into 5 distinct stages.

1. Product Strategy
2. Product Design
3. Product Development
4. Product Marketing
5. Product Maintenance

Each of these stages by itself can offer unique competitive advantages, but the key for success is to be good in all five stages. To understand this better, let me explain each of the five stages.

Stage-1: Product Strategy

At the highest level, Product Strategy addresses the question: Why are we building this product?

This is the initial phase that starts off with the basic rationale for the very existence of the product. Starting with a well defined problem statement and being thorough and accurate about the reasons for developing the product provides a stable platform upon which one can build a successful product.

Since the product will evolve over a period of time, it is important to have a roadmap to achieve the product goals over a period of time. The roadmap defined a set of steps that will take the product from start to its intended goal - which answers the question: Why are we building this product?

Having a well defined product strategy itself provides a solid competitive advantage - as it gives a solid foundation upon which one can take quick decisions.

When defining the strategy, the starting point must be: How will this product will bring in revenue?
Having a clear revenue model at the start will help immensely all through the product life cycle. The revenue models will help in product design and product development.

The second question is to address the primary purpose of the product: What customer problem is it solving? The primary purpose will define how the product is designed, how users will interact with the product and what the user experience with the product should be?

Answer to this question will determine the distinct set of features, functions and requirements to considered during product development. This will also help in organizing all the features, functions and requirements for subsequent releases and help create a product roadmap.

Third question to ask is: Who is our competition?
Having a complete competitive analysis will help understand the market better and also learn from mistakes done by competition, so that this new product can have strengths where the competition is weak. Doing a competitive analysis will help avoiding reinventing the wheel. If the competition is doing something that works, then it is best to copy it.

There are three outputs from this stage of defining the product strategy:

Output-1. High Level Goals
This consists of the best possible business model for the product. It also defines the success criteria for the product. The business goals must be simple enough for everyone to understand and must not have any contradictory goals.

Output-2: Define the customer problem which the product will solve.
Create a statement of the problem(s) or need(s) the product will solve for customers. Having the problem/need and solution properly will provide valuable guidance for product development teams.

Output-3: Competitive Landscape. 
Doing due diligence on competitive landscape will help cut down the development time and avoid reinventing wheels. Knowing what works in the market will help big time when developing new products.

Product Strategy forms the solid foundation in which the success or failure of the product rests. One cannot afford to by-pass strategy definition stage and hope to succeed.

Stage-2. Product Design

Once the broad product strategy is defined , the next most important stage is to come up with the product design

At the highest level, Product Design addresses the question: How are we building this product?

Once the strategic foundation for the product is defined, the work on product design has to start. In this stage there are two very important questions that needs to be addressed:

1. What should be the user experience?
2. What technology platforms must be used?

One must take a wider view when considering the user experience of the product. It must include all aspects of user experience such as User interface, functions, product performance, product stability, reliability, etc. It must cover all aspects of user experience from the day of first usage to its eventual retirement. For example, Apple iPhone is designed with the user experience from the moment one walks into the Apple store, includes the packaging - the experience of opening the product package, the usage experience, and even product return.

The goal of product user experience is to be "Awesome". It must be this word that customers will use to describe your product.

Product Technology Platform is the next critical aspects that needs to be defined. In today's rapidly changing world of technology innovation, having the right technology platform makes a BIG difference.

What denotes the right technology platform depends on the product segment. For software it may be a set of underlying software development tools and development framework and the underlying OS and deployment methods etc. For computer hardware, it could mean a x86 processor or ARM or Qualcom etc.

The underlying technology framework must address questions such as:

1. Do we have the necessary skills in house to use this technology?
2. How does this technology offer intrinsic advantage?
3. Does this technology provide all the functional elements needed in the product?
4. Does this technology allow the product scale or perform at the levels expected by power users?
5. How does this technology help in the over all  design & architecture of the product & subsequent portfolio of products?
6. How does this technology help the product fit into the customer ecosystem?

The most important aspect of design stage is to ensure that people with right design skills are working on it. Often times not having the right skills hampers the design process and severely handicaps the product - which will result in product failure.  It is best to hire external consultants during the design stage and get the best of design talent.

Selection of underlying technology platform is a core engineering function that defines the product's user experience from start to finish that can eliminate problems for both the developer and users for a long period of time as defined in the product roadmap.

At this stage, Product design is not about the actual development of the product - but it is defining the tools and the framework upon which the product will be built, supported and sustained.

Note that doing competitive analysis and market analysis before embarking on product design stage can benefit immensely - as one can learn from competition and market landscape in choosing the right technology framework and also defining what the user experience should be.

Knowing what the customer expect from existing competition and their current user experience will help in a big way to create a better product.

There are few best practices that are generic enough to be applied across different products. Some recognized best practices are:

Take time to complete the design. Avoid the rush to start development as soon as possible. Instead make sure that the designs are complete. This will save time, effort and big costs later in the project.
Long time ago, when I worked on a SoC project at Silicon Valley, we spend nearly 25% of the project time in the design phase. Every aspect of design was completed, the choice of CPU cores, Memory modules, and other hardened IP was first determined, even the power envelope for the product was designed and calculated, the overall design layout of the chip was almost finalized before the actual development started. The result was a blockbuster product.

Document the Design. Spending time in documentation of the design, the requirement analysis, functional features etc. saves a lot of time during implementation. The design document serves as the product blue print.

Cut all dead wood.  Typically while designing new product, there will be demands to maintain some legacy aspects for backward compatibility. While some of the legacy requirements make business sense, some do not. Review all the legacy requirements and eliminate those which does not make business sense. This might take some bold design decisions. For example, Apple's decision to eliminate floppy drive in iMac or eliminating DVD drives in MacBook Air.

Consider the overall product ecosystem for the product. The end product must fit into the user ecosystem. Ignoring the strengths of the existing ecosystem can hamper the success of the product.
Involve  the development team. Make sure that the development team is involved in design process as early as possible. This will help them prepare and develop the product faster and better.  Product development team can provide valuable insights from start to finish and can eliminate problems.

Involve product support & customer service teams in design stage. Getting inputs from product support & service teams will help design the product for better serviceability. For example, developing software with better debug messaging will help resolve customer issues faster when the product is deployed at customer site.

Additional best practices can be learnt from experiences of other companies & competitors. There is no shame in copying and learning from other industries either. Learning from others helps lower costs  & time taken for product development.

Stage 3: Product Development

This stage answers the question: "How do we create this product?"

Once the product design is complete and design is documented, the next stage is to build the product.

During the development stage, there are three critical factors one need to consider.

a. Customer Feedback and Acceptance
b. Development Program Management
c. Product Testing & QA

a. Customer Feedback and Acceptance 

It is a good practice to involve a customer in product development stage. I have written a detailed article on co-development with customer See:

Getting customer buy in on the usability of product functions, features and user interface during the development time speeds up customer acceptance of the product and lowers the cost of sales. Developing products with constant customer interaction helps, but one also needs to know where to draw the line when it comes to meeting customer demands. For example, Google runs a long Beta program during which it collects lots of customer feedback data and tunes the product accordingly.

Ideally, the Beta program need to be designed into the product development process - so that it can be managed for optimal outcomes.

b. Development Program Management

Program management is another vital aspect of product development. No product development project will be successful without a strong program management. Program management plays a critical role in providing leadership, helps coordinates various development teams and ensures timely results.

In product development, there will be several different teams (development, release management, legal, documentation, marketing, finance, etc.,) working on the product and these teams could be globally distributed. So it is the role of program management to coordinate between the teams and provide arbitration between teams to ensure timely outputs.

Program management brings various stake holders together and facilitates decision making and in cases provide leadership to product development teams.

c. Product Testing & QA

Quality is a default expectation today. Competition will ensure that any slips up in quality is punished. So there is no room for error or slips in quality. In many cases, customers can also sue the vendor over defective products, or regulatory agencies can levy heavy fines. Thus slipping on quality is non-negotiable.

Another aspect of quality is performance. Product developers must keep the user experience top of mind and ensure performance of the product is not compromised.

All this implies that one needs to invest heavily on quality assurance process and testing.

However, the pressure for rapid product development and the need to shorten the development cycle is leading to cut backs on testing - but will always boomerang in form of product failures, higher costs of product support and need for bigger sales/marketing budgets.

In my opinion, it is better to spend a dollar on QA first - rather than spend the same dollar on customer support issues or product marketing.

Stage 4 – Sales & Marketing Strategy

As the product is getting built, one needs to answer the question: "How do we sell this?"

Today, markets area already over saturated with products. Even if there is a real customer need, getting the message to the customer is turning out to be a big challenge. Product sales strategy is essentially a plan that addresses this challenge. While sales and marketing departments have to operate in close cooperation with product development - mainly because there are several sales/marketing decisions that impacts product development:

1. Product Release schedule
2. Product Distribution Channel
3. Product Bundling
4. Product After sales support
5. Product Training
6. Product Pricing
7. Product Revenue Forecast
8. Actual Product Revenue.

There are whole books being written on sales and marketing of products and this article cannot do any justice in giving a complete sales & marketing strategy.

Sales strategy has huge impact on product pricing and product release strategy, So defining the sales & marketing strategy at the time of product development is absolutely vital.

Stage 5:  Product Maintenance & Customer Support – 'Making It Sticky'

In an ideal world, customer just buy the product and use it without any problems. They never have to contact the manufacturer and keep buying more and more of the product. But in real world, businesses must constantly elicit feedback and help customers use the product. This is the function of product maintenance and support.

Customer support strategy often serves as the feedback loop to the overall product strategy and product design. Issues found on-site has to be fixed and addressed in the next release of the product. Having a solid product support groups provide constant input to all business units involved, helps develop better product.

Keep your customers engaged with a seamless experience that makes your product stick. This will make your customer comeback for repeat purchases.  To ensure a seamless experience, one needs to:

1. Educate customers on what the product does. How your product is different than competition? How your product does things in a new & Novel way?

2. Ensure product performance on customer site is as per the product's promise. There will be differences in your test/dev environment and the real customer environment. Customer support team will have work with customer to ensure that the performance, stability & reliability of the product is as per customer expectation ( customer expectation is set by marketing)

3. Ensure product security. This is more applicable to software products or software related product delivery. With changing security threats, one needs to constantly update the product to address all product security issues in a timely manner.

4. Collect feedback from customers and channel partners on how the product is performing in the market, also collect feedback on product inventory and sales velocity.

5. Product Support groups often form the front-line when it comes to communicating with customers. Communicating with customers helps making their experience better and more rewarding, thus adding to product value.


Building a world class product is not a one time activity. It is a continuous process with several interlinked stages and constant feedback between stages

The eventual success of the product depends on strengths of each individual functions AND the strength of the interlinks between these stages. Stronger the interlink the better the product. The strategy for developing world class products is not rocket science - but the execution of all aspects of product development is!

Tuesday, November 03, 2015

Building a Leading Product

When a new product is introduced, it is extremely rare for a product to become a market leader right away. Most often, products take time to build a market share and become a market leader.

Building market leading products can be drilled down to a process. If product management executes on this process, one can build a market leading product. Building leading products is a joint function of engineering and product management.

Looking more closely at all the leading products, I have distilled this into a few product management characteristics that can serve as valuable guideposts.

1 . Know customer context and develop products accordingly 

Initial product development, say for version 1.0 is often based on certain assumptions on how customers will use the product. But when the real product is released, and customers start using the product, the actual customer usage will differ from the initial assumptions.

One has to develop a deep understanding of customer context - on how, where, when, why, what - they are using the product for, and that becomes the basis for the next round of product improvements  in the next release.

Understanding how customers use the product is the key. This insight can be gained by mainly by observation and customer interaction. I call this as Customer Anthropology.

Studying customers using the product gives a deep understanding on the context of product usage and product limitations & issues faced by customers.

2. Take a broader view of the Customer Experience

It is also important to observe (potential) customers who are not using your product and gain a deep understanding as to why they are not using your product - and instead they opt to use a competitive product.

Customer experience and use cases are very diverse and one need to look in detail on what the actual customer experiences are. Often time, it helps to think in "What-if" lines:

What if the product responds more quickly?
What if the product does this additional function?
What if the product costs more? (or less)

Taking a broader view of customer experiences will give the deep insight on how they can beat the competition and emerge as market leader.

 3. Act on insights systematically 

Once you have gained the deep insight on how customer use your product, and also why certain customers don't use your product, product leaders must act fast and systematically to address the product shortcomings.

This sounds a lot easier then actual reality. But most companies fail to act in their insights for various reasons and let the product fail.

Acting on insights takes real leadership. Ability to influence all stake holders and drive them to act consistently and systematically is the key to success!

Monday, November 02, 2015

Can NetApp Survive the EMC-DELL Whiplash?

All data storage vendors are going through a tough times. EMC decided to sell itself to Dell and go private to help survive the coming storm. Ever since EMC-DELL deal was announced, investors and customers are looking at NetApp - wondering if NetApp will survive as an independent storage vendor.

Enterprise storage industry is in a state of transition that is driven by the confluence of multiple technological advancement including Flash, software-defined storage, Big-Data, Cloud, and converged/hyper-converged systems.

Given the tough business environment, NetApp is going through tough times. Q3 2015 revenue dropped by 11%. As revenues drop, free cash flow has also dropped for last three quarters in a row. Eventually, the bottom will fall off, and profits will disappear and blood bath of red ink will spread all over NetApp's balance sheet.

NetApp Insight Conference in October 2015 indicates to that NetApp is trying to reverse the sales decline - mainly by migrating customers to 'Data Fabric' and aims to reposition NetApp as a global data management company.

Data Fabric strategy is designed to allow customers to store, access, protect, share and archive data in consistent and predictable ways across multiple internal and external data centers - including public clouds.

The days of NetApp selling storage boxes will soon come to an end, and the new CEO George Kurian has a big task on his hand to transform NetApp into a Storage Software Vendor for Hybrid clouds.

This denotes a major shift in NetApp's strategy  - which is fraught with risks.  Along the way NetApp will continue to lose revenues, it will have to rejig its sales force who will have to work hard to convince its existing enterprise customers to move to data fabric.

When revenues drop so does share prices, and with that there will be large attrition and job cuts. With declining revenues and reduced staffing, the strategy will be tough to execute.

Challenges Facing NetApp

NetApp is facing several challenges - which can be summed up in one word "Google".

Not that Google is a direct competitor, but it personifies all the challenges NetApp faces:

1. Data moved to Cloud
2. Move towards commodity hardware
3. Open Stack & Open Source Software

Data moved to Cloud

Cloud is now main stream, and customers are moving lots of data to cloud. Recent advances in cloud archive storage, cloud access gateways and hybrid on-premises/cloud data management will accelerate the movement of data to the public cloud.

As enterprises move data to cloud, they will need less of storage boxes - which has a direct impact on NetApp's revenue.

Move towards commodity hardware

Big Cloud service provides have built their data centers on commodity hardware and have also developed deep engineering expertise to design and build complex storage systems for their data centers. As a result, none of the big cloud service providers (Google, AWS, Microsoft Azure ) are buying storage boxes from NetApp.

To rub salt on NetApp's wounds, Facebook and others are supporting Open Computing Project - which publishes design specifications and design documents for the custom-built servers, racks, and other equipment used in Facebook's data centers. Other cloud service vendors are fast to copy suit and order generic hardware ODM vendors in Taiwan and Asia.

Eliminating the requirement for proprietary hardware and embracing off-the-shelf platforms is leading the next revolution of data center technologies, and this is no place for NetApp.

Industry's move towards commodity hardware and Open Compute Project is rapidly eating into NetApp's revenue, and thus taking out steam out of its engine.

Open Stack & Open Source of Software

Open Source movement that started in universities has become main stream and is now impacting hardware. Starting from FreeBSD and Linux, Open Source software has grown to offer entire cloud stack solutions - called as "Open Stack". Open Stack software allows companies such as Google, Rackspace to build cloud scale data centers on commodity hardware.

Today, several companies including IBM & HP are supporting & distributing Open Stack - which is a complete set of software solution needed to run a data center. NetApp also taken a knee jerk reaction to integrated its arrays with Open Stack.

In addition to Open Stack, several other vendors are open sourcing their storage software. EMC recently announced open sourcing of its ViPR Controller - a Software defined Storage solution. Similarly Nutanix made its Acropolis App data fabric software free and open source.

The steady flow of open source software from established vendors makes it tougher for NetApp to sell high value data fabric software stack.

Open Source Software movement adds tremendous head winds to NetApp's sales and transition to "Data Fabric" strategy.

NetApp's Missteps

Internally, NetApp faces several challenges. Many of them comes from its strategy missteps and NetApp missed several good opportunities:

1. Missing Flash in Sales
2. NetApp misses the BigData boat
3. NetApp slips on SDDC & SDS solutions
4. Share buyback instead of technology acquisitions

Missing Flash in Sales

EMC was a latecomer to Flash party.  But in a short span of time, EMC's ExtremeIO became the market leader with more than 30% market share. Comparatively, NetApp's Flash Array lost out to ExtremeIO and Violin Memory, a startup.

Even now, NetApp does not seen to have a good product roadmap or product strategy to conquer Flash Array market.

NetApp misses the BigData boat

BigData needs Big Storage. Apache HDFS is designed to run on commodity hardware using the disk drives connected to servers. However, enterprises found that there is a need for a dedicated array for low cost data storage needed for Big Data Analysis. While EMC executed brilliantly with its Data Lake strategy - positioning EMC Isilion and EMC ECS arrays, and with Pivotal HD distribution. Today, EMC Isilion alone generates more than $2 Billion in sales from BigData sales. (Also see: Data lake - Solving the challenge of Big Data Integration)

NetApp slips on SDDC & SDS solutions

NetApp missed the Storage virtualization and Software Defined Storage market and waited too long to respond. As a result EMC & VMWare are able to take a lead with VSAN, ScaleIO and ViPR,  & ECS in large enterprise market, while several startups such as Chep, Nutanix, Nexanta have also leaped ahead.

In response, NetApp released FAS 8000 Flexarray Virtulization software - which was still tied to its hardware and did not get much success. Though, there is a lot of work in progress within NetApp on SDS - but I guess that NetApp has missed the boat.

Share buyback instead of technology acquisitions

One Big Strategy which worked for NetApp in the short run was Share buyback. NetApp constantly rewarded its investors with big share buybacks. $1 Billion in 2013, $1.2 Billion in 2014 & $2.5 Billion in 2015.

These big share buyback programs helped fend off pressure from Elliot Management, it also meant that NetApp did not have the money to buy other technology based firms. As a result, NetApp has fallen behind the technology curve in all aspects of storage technologies.

Since 1996, EMC has acquired over 50 companies, while in comparison NetApp has completed only 9 acquisitions.

Road Ahead

Transformation from a hardware vendor to a software vendor will be a work in progress for a long time. During which NetApp will have to rejig its top management. Departure of the company's CMO, Julie Parrish, is just the beginning.

Moving to Data Fabric Strategy entails selling more software to its existing (& shrinking) customer base. One hope for NetApp is that customers will move back to some on-premise solutions after experiencing 'sticker shock' from AWS and other public cloud services. According to Val Bercovici, NetApp's 'cloud czar' says this process is already under way.

Another hope for NetApp is that Dell-EMC deal will distract EMC for a long time and this will allow NetApp to create a space for itself. In addition, splitting of HP & Symantec-Veritas also helps in distracting competition.

But a multi-billion dollar business strategy cannot be based on HOPE!

My Take on NetApp

NetApp is one of the 'old guard' stand alone storage vendors left in the market. With an impending revenue collapse and severe market challenges, there is no way NetApp can survive on its own in this environment.

NetApp's product positioning of 'Data Fabric' as a bridge between private infrastructure and the public cloud has merit but it has huge market challenges. NetApp was never known as a software vendor and to succeed as a software vendor, it needs to merge with a strong software player - such as Microsoft or Veritas, who can take on a smaller and focused market.

I doubt Kurian and his management team can transform NetApp to become a storage software company while protecting its revenues. NetApp must strive to protect its margins - while it jettisons its hardware business.

A good option is to spin off its hardware business, while merging its data fabric software with another enterprise software vendor. In short, Dell's EMC acquisition rings a loud bell for NetApp, and like EMC, NetApp's days as an independent company is numbered.

Thursday, October 29, 2015

Most Common Leadership Failures in Startups

Recently, Mr. Mohandas Pai, CEO of Manipal Learning and former CFO of Infosys made a statement that most startups in India will fail. (see: Only 10% of startups will be very successful)

While it is a general truth that 90 percent of startups will fail. Founders often attribute business failure to a range of issues that range from a lack of capital, poor marketing, ineffective product development or the inability to meet demand.

But the real cause for failure is likely to be Leadership.

Leadership, or lack of leadership is a BIG problem startups face. Its not just leadership at the top level, but also at all levels of the company. Leadership issues in the organization eventually boils up slowly leading to implosion of the business.

Here is the list of five most common leadership mistakes:

1. Expecting too much, too soon.

A typical startup is always in a hurry. Projects are not properly planned and demands are constantly changing and adding immense pressure on teams to deliver. This adds undue stress on the team and the team delivers an incomplete or defective products. Expecting too much too soon just doesn't work.

Solution: Leaders must take time to get a clear understanding on expectations and must not be afraid to say "No" when it is not possible to meet new demands.

2. Not providing adequate resources

It is typical of startups to demand something from nothing.

All organizations need: resources, time, and clear requirements. Often times in a startup resources are in critical shortage, this implies spreading the workforce too thin. This will work only in short term - but it also impacts quality of work. Adding unnecessary pressure to an already frenetic work pace does not result in great products.

3. Failure to speak the business language

Startup is not about a product or a service. It is a business that runs on money or cash. The language of business is money. Most technocrats who start companies are not comfortable  in communicating in real numbers of money. Often they end up using "soft terms" instead of hard numbers - which over a period of time leads to poor decisions and eventual failure.

4. Leaders hire & reward conformism.

"We are a great team, we think alike & we work alike." This seems to be a common motto in most startups. Often times, startups are usually all men, in the same age group, with a very similar background and experience. Group thinking often prevails, and that often leads to failure. Anyone who fails to confirm to the leader's thoughts are thrown out.

The reality is that great minds do not always think alike and if one could work effectively with people who have different point of view, and their value is judge on the merit in very important. When you are hiring or promoting someone - first think 5Ws and H questions and then make the decision.

Who is being promoted?
Why is this person being promoted?
What are we promoting, the person or the skill?
When is the opportune time to promote this person?
What will be the obstacles to his or her success?
How can we help him or her? 

5. They don't communicate bad news

Communication is at the heart of getting results. Good news or bad must be communicated. Bad news in particular must be communicated with great care & clarity.

Down playing bad news or sugaring up failures in town hall meetings is a very common mistake. As a result the team does not know how to adjust to the actual reality and that in turn leads to poor decisions.

Leadership at start ups have to be very open when communicating internally with all employees, and that clarity which comes from effective communication helps everyone to take better decisions.

Sunday, March 29, 2015

Data lake - Solving the challenge of Big Data Integration

Big Data analytics is a game changer for businesses today. Unfortunately, most organizations are struggling with collect & integrate vast volumes of data needed for business analysis. As a result, with poorly integrated sets of data undermines business analysis and executive decision making process.

As organization start to implement their big data analytics projects, the first step is to develop a comprehensive strategy for managing data:

  • A strategy that should incorporate all sources of data needed for analysis. 
  • A strategy that should incorporate capable technology & tools for big data
  • A strategy that make data integration in a smooth & fast to provide timely analysis.

Companies that are well equipped for big data integration will operate more efficiently and effectively. Data lakes enables companies with new generation of technologies - which is the first essential step to increasing agility in business.

The challenge

Organizations are seeing huge increase in volumes of data. Data is coming from various sources:

1. Structured data from databases, web pages, OLTP, etc.
2. Employee created unstructured data in form of files, emails, IMs, etc
3. Machine generated data from sensors
4. Video surveillance feeds
5. Misc. user generated data: Photos, videos, pdfs etc
6. Data from external feeds: Social networks, Twitter, news sites, web comments etc

As types & sources of data increases, the challenge of data integration multiplies. The traditional data warehouses cannot cope with new types of data and is not designed to handle this high volume and variety of data. As a result, the traditional BI tools fail to give meaningful insights for decision making.

In the world of big data, Legacy BI tools are slow and error prone.  There is a widespread dissatisfaction with their current data integration technologies and organizations are finding it too slow and hard to maintain data.

According to a study done by Ventana research:

  • 78% of organizations are facing challenges in integrating different data sources.
  • 55% of companies are somewhat confident or not at all confident in their ability to process lage volumes of data
  • 58% doubt their ability to process data that arrives at high velocity. 

Organizations waste significant amounts of time on data integration tasks, particularly in reviewing data for quality and consistency, which is needed to prepare it for business analysis.

Data integration must be fast and accurate for market place agility. Most organization need data on hourly or daily basis. In Internet economy, real time data analytics is the key for success.

It is critical that data integration and data ingestion capability to be flexible enough to deliver multi cycles of processing to satisfy different analytical needs - i.e., to be used by wider big data analysis.

Use of public cloud for applications also complicates data integration. Organizations are having a mix of public cloud and on premise IT - which essentially complicates data integration and timeliness of data for analysis.  Accessing data in traditional batch cycles is not the best way to utilize cloud data sources.

As a result, companies are looking for tools to automate data integration.

EMC Data Lake Foundation

EMC Data Lake Foundation with Pivotal Suite of Big Data analytics can address most of the data integration challenges.

EMC Data Lake Foundation - which is based on EMC Isilon and EMC ECS (Elastic Cloud Storage),  integrated with the rich analytics tools from Pivotal can provide a common integrated data pool  - thus make it simple to collect, store and  analyze massive volumes of data.

EMC Data Lake Foundation solves the problem of data integration by providing a common data lake that accommodates both high velocity unstructured data, machine data and tradition databases. With Pivotal suite of analytic tools, while leveraging existing BI tools in the mix allowing existing business analytics to work along with new Big Data analytics.

Unified Data Lake is the game Changer

Creating a unified Data Lake which had ingest and hold both traditional data in existing data warehouses and newer data types should be the first step while embarking on a big data journey.

A unified Data Lake gives companies a choice of data extraction and analytics tools and does not lock workers into using old existing solutions, Older workflows can be easily integrated with newer Big Data analytics workflow.

A unified Data Lake allows new Big Data analytics solution can use  new technologies like Hbase, Storm, Hive, Pig, Mapreduce, Gemfire, etc to provide analytics for different applications, while providing enterprise class data security, protection and access control in a centralized, integrated way that data is accessible and easily managed.

A Unified Data Lake offers several benefits including:

  • Agility: Eliminating much of the strain on IT that was common with traditional silo approaches.
  • Simplicity: Allow consumption of data in any format, thus saving time & reducing errors
  • Flexibility: Allow for the use of different analytics techniques, mix of both old and new, which helps organizations see the data differently to ask new questions and derive new insights
  • Accessibility: Provide users with fast, easy and secure access all their data.

Closing Thoughts

A well thought out data integration strategy with EMC Data Lake foundation will enable companies to reap the full benefits of Big data. The data lake allows companies to:

1. Retain and analyze more data
2. Increase the speed of analysis
3. Secure business data with enterprise class data security systems
4. Meet business needs for decision making
5. Make more information available across organization

Integrated data lake will maximize the return on investments in big data analytics.