Product Management: Manage Requirements for Project Success

India has been a major power in computer software, but lot of companies struggle to develop software products. I have seen a lot of start-ups struggle with new product development and after several months or years of efforts product development projects are canceled.

The major problem with Indian IT companies is that they lack proper product management skills and fail to properly manage the product requirements.  In this blog, I have listed the basic steps in managing requirements to ensure new product development success - particularly for start-ups.

All product development starts with identifying product requirements. The job of product manager is to define the product requirements and work with engineering and project management office to implement it.

Successful new product development is not an easy project and there is always chances of mistakes or missteps. From experience, I have noted down few mistakes that must be avoided in all projects.

The most common problem is that of project overruns. Often the project costs more than the initial estimate and/or takes more time to complete. To avoid getting the blame for the overrun, some teams try to release a half baked product. Which results in a much bigger problem of fixing the issues after the product release.

In this blog, I have listed the basic steps in managing requirements to ensure project success.

1. Document the minimum set of requirements that has to be met to ship the product.
2. Avoid misinterpretations of product requirements.
3. Do not 'Overbuild'
4. Define 'quality parameters' as part of product requirements
5. Define compliance as part of product requirements
6. Keep a close eye on project team coordination
7. Build an agile product development process
8. Avoid excessive changes to product requirements
9. Ensure Executive involvement
10. Ensure a Beta test program with a real customer.

Document the minimum set of requirements that has to be met to ship the product.  

Before starting the development project, the first step is to document the requirements in ways that can be clearly understood by the team. If needed, use visuals or animation or mock-ups to make people understand the requirements.

These requirements must also be prioritized into "must have", "Good to have",  "Bonus" etc. The final product cannot be released if it does not meet the "Must Have" requirements. Once the "Must Have" requirements are identified, then work on estimating the time & resources needed to complete the project.

Product managers & all stakeholders must review the product to check if all the must-have features/requirements are met before proceeding with the product release.

Avoid misinterpretations of product requirements

Understanding product requirements is always a challenge. Often times, the requirements are documented in plain text. Which can be misinterpreted, leading to chaos/confusion at the time of product release.

There are several levels of requirements that must be documented. Starting at the high-level of the product concept, Architectural details, and down to user interface design requirement.

The best solution is to document the requirements using visuals - either in terms of visuals or animations or mockups. In addition, one can create user personas to illustrate the end user requirements.

During the development process, ensure that the entire product development team has a common understanding of the requirements.

Do not 'Overbuild'

A common problem with building new products is to add loads of bells and whistles into the product or build capacity/capability which is far in excess of user's real requirements.

Another manifestation of this problem is to add too many features. In most cases, there is just one customer who needs that particular feature!  This leads to adding too many features which confuses customers.

Overbuilding a product costs money, time and resources - all that for features that customers do not want!

So care must be taken to ensure that the final product actually has what is needed.

Define 'quality parameters' as part of product requirements

Quality of the product must be defined in the product requirements. The quality is as perceived by the customer. For example, Mean Time Between Failure (MTBF) of the finished product must be defined in the product requirements. In case of software, the system stability, or memory growth must be defined in the requirements. The quality parameters such as "response time" or "resource utilization" etc. must be defined as part of the product quality.

Define compliance as part of product requirements

Just like quality parameters, the product compliance requirements too must be documented in the product requirements. Often times, I have seen that the requirement document makes only a passing reference to compliance adherence - and does not dwell deeper into the actual technical details of the requirements. Such high level description is a perfect recipe for disaster - as it opens up the field for misinterpretation of requirements.

Keep a close eye on project team coordination

Today, distributed project teams have become a norm. Even startups out source some segments of projects. In such cases, the team coordination becomes a major hurdle, leading to miscommunications and project over runs. Any signs of poor coordination is a red alert on things going wrong in the project.

As part of Product management oversight on the project, it is important to keep a close eye on the team coordination and take proactive steps to improve team coordination. Product management must work with project management and engineering teams to constantly monitor team coordination and take corrective actions as necessary.

Build an agile product development process

In today's rapidly changing world, it is important to be agile - not just in project management, but also in Product management. Agile product management helps in modifying the end product to meet the market needs - even when the project is under execution.

Agile product development allows product managers to fine tune product requirements and meet customer's changing requirements.

Avoid excessive changes to product requirements

While agile product development helps, excessive changes are detrimental to the product development. Especially, when changes are made to sections of the project that is already built/coded. So any change in product requirements must be validated and vetted by all stake holders. As a thumb rule, a project should not have more than two changes to requirements - to sections that is already completed, and any change must be approved by the change request management committee - a.k.a. stake holders.

After the second change request, any subsequent change requests must have an executive level approval.

Ensure Executive involvement

New products are part of the overall business strategy. So the top executives must get involved in new product development and provide strategic inputs when required. Top executives must also oversee the project and provide active project governance. In case of startups, when the founders themselves are deeply involved in the project - the board of directors must provide the higher level governance.

Ensure a Beta test program with a real customer.

Beta testing the new product with the customer solves a lot of problems in the product development stage and also improves customer satisfaction and user acceptance. Getting customers to test the product before release will help uncover functional bugs, design gaps and other user environmental issues which could not be found in the internal testing cycle. Often times, developers use simulators for testing - which does not reflect the actual customer environment. So the Beta test helps identify those bugs or even design flaws - which can be fixed quickly.

Remember: the cost of fixing a bug in beta testing is lot cheaper than fixing it after the product release.

Closing Thoughts

Project success is also dependent on managing requirements. Product Managers must keep a tight control on product requirements. Many Indian software startups try to build a perfect product and try to please as many customers as possible or try to add too many features/requirements. This leads to an unwieldy projects - which eventually fails.  In this article I have described 10 main points product managers must follow to ensure new product development project a success.

What Marketing wants from IT

Today marketing technology products is fully digital. Product Marketing is a big user of big data analytics. The potential benefits of data analytics for marketing is immense. Marketing has been an early adopter of cloud SaaS services www.salesforce.com.

In this business environment, marketing needs more than just a plain vanilla tech services from IT departments.  To succeed, CMOs need more that the traditional application hosting, support, security services. CMOs need IT departments to be an active partner and help leverage technology to develop cutting edge marketing initiatives such as predictive analysis, creating custom offers etc.

This new jobs will be in addition to the current role of keeping the infrastructure running and the systems secure. As a product manager, I am involved in several product marketing initiatives and I interact with several CIOs. These interactions have given the insight into these new requirements.

1. Enable Big Data Analytics

The IT systems are collecting tones of data. But this data is locked up by IT in different silos. Marketing needs to access this data in real time to run data analytics. CMOs need IT to provide access to these multiple data sources in an integrated way - i.e, provide a quick, safe and secure access to the data lake to run their data analytics.

Managing, collecting and making use of internal and external data is one of their top five challenges, according to more than 500 marketing professionals who responded to IBM's 2013 Global Survey of Marketers.

2. Help to measure new KPIs

Social media, viral marketing, multi-channel customer engagements, and big data analytics are creating new key performance indicators, or KPIs. CMOs understand the value of these new KPIs but often struggle to measure it.  Marketing needs help in measuring parameters of time taken at the billing section, time taken for servicing a customer request etc. Getting these new KPIs helps marketing to optimize & improve all customer interactions and deliver a truly outstanding experience that matches the company's brand promise.

IT needs to understand the new KPIs and then help marketing by building IT systems to measure it. Thereby helping marketing to succeed.  This implies that IT must go beyond the back-office functions and become an active technology partner in marketing activities.

3. Help to remove shadow IT & manage technology expenses

Marketing works at a pace which is much faster than what their internal IT can support. So often times, marketing chiefs create a shadow IT - by outsourcing some IT function - such as creating & hosting specific websites, or subscribing to a SaaS or PaaS or IaaS services. While outsourcing & creating a shadow IT helps, it also costs more. So over a period of time, the cost of maintaining shadow IT becomes prohibitively expensive.

CMOs want their IT departments to eliminate this shadow IT by bringing back the external hosted sites/applications in-house and by consolidating all the outsourced IT services.

According to Gartner report, by 2017, CMOs will spend more on IT than will CIOs.  So CMOs want IT to bring its technology expertise, knowledge of existing infrastructure and newfound understanding of marketing's objectives/KPIs to bear in helping marketing make the best investment decisions.

CMOs want IT to guide/assist/help in selecting technology platforms and provide data on viable IT systems that meets the business needs.

4. Faster Turnaround time please... 

One reason why CMOs use shadow IT is due to the fact that their own internal IT department cannot service them in a timely manner. Today marketing campaigns are planned and executed in days. For examples a road show is planned, and a web site has to be created for the event in 3 days. The CMO approaches IT with this request, but IT needs 4-6 weeks to test and host the web site! Such a long turnaround time is unacceptable for CMO and they end up outsourcing to a third party or signing up for a cloud service.

5. Enable a faster pace of  Business

Smart Phones & Tablets have changed the pace of marketing. Customers now have access to Internet from anywhere anytime. Added to this mix is the mobile social media: Twitter & Facebook. This implies that customers are interacting with the company in new ways - which require faster response from marketing departments.

Marketing now has to handle all customer interactions -- outbound, inbound and those happening on social media in real time - and for that they need new technologies that allows them to interact with customers at any time in any of those media.

But the IT departments are still hesitant to support the new mobile platforms and insist on doing things the old way for security. For example, IT does not allow employees to carry corporate data in mobile fileshare app such as Dropbox.  In many organizations, IT is yet to embrace Virtual Desktops on Tablets.

This slows down the pace of business & CMOs want their IT departments to change from being a brake on business wheels to a booster rocket.

6. Create an Office of Chief Marketing Technology 

The fast changing landscape of technology - with mobile, cloud, Big data and IoT, it has become a necessity for marketing departments to have a specialized technology office - which looks at new and emerging technology and determines how/when/where to use these new technologies. Traditionally, new technology evaluation and validation was done by IT departments - but now CMOs are creating an Office of Chief Marketing Technology.

Chief marketing technologist is responsible for understanding both available and emerging marketing technology platforms and articulating how marketing can capitalize on them. They play a strategic roles and some cases tactical roles in terms of getting involved in procurement, or setting up marketing technology centers of excellence etc.

In many organizations, marketing departments lack the technical skills needed to create the Office of Chief Marketing Technology and they need help of their IT department to set it up.  In some of the large fortune-500 companies, the Office of Chief Marketing Technology also works with IT to run & maintain new technology platforms.

7. Let marketing run its own software systems 

To reach out and communicate with customers in real time, Marketing departments have stated to run their own software. New technologies such as big data analytics, Internet-of-things, Mobile apps etc. has created several new requirements - where Marketing wants to have hands-on control on the software systems, while IT runs all the backend infrastructure.

In this new paradigm of mobile & Cloud Apps, Marketing want to own and control customer experience by running the software - where marketing team can quickly alter design, develop a new user experience or create a new technology-driven marketing initiative, without doing lots of coding or fixing bugs.

CMOs want IT to empower marketing in how they interact and deliver personal experiences to customers. In this way, IT builds the backend infrastructure and creates an engagement layer - which interacts with marketing department to empower technology users.

For example, IT creates & maintains a big data platform - which marketing department runs to get real-time insights.

Closing Thoughts

Big data, social media and mobile technologies are shaking up the old world order. CMOs are eager to bridge the big gap between traditional marketing and IT to take full advantage of new technology driven opportunities for better customer engagement and higher ROI.

Some of the leading organizations have created shadow IT or mini-IT within marketing departments, but then they are now realizing its limitations and challenges.

The right solution is for IT to take a proactive steps and work with marketing departments to bridge the technology capability gaps and lead the transformation of how marketing teams engage customers.

Also see:

1. Understanding Big Data
   
2. Defining the Business Objectives for Big Data Projects
   
3. Understanding Data in Big Data
   
4. Next Technology Disruptor: Internet of Things

Rise of Anti-Social Apps & Limits of Social Networking

Smart phones have indeed made people smarter. Right now, there are millions of Smartphone users who are worried about their loss of privacy.

On one hand Smart Phones gave users tremendous power to communicate with anybody in real time - via social networks. Social Networking Apps such as Facebook & Twitter gained immensely from this. But with passage of time, users are slowly understanding the "dark" power of social networks: Loss of privacy. Some people found it the hard way - when they were not getting hired - because of some "party pictures" in the social network, or a comment.

The social networks help people connect and communicate, but it also remembers all communications forever, and all communications are instantly made public!  This leads to loss of privacy.

Smart phones today have built-in GPS and it can imbed all the locational information in photos, videos, messages etc. Cell phone service provider can track the location of each smart phone using the GPS or by triangulation. In extreme cases, malacious software can be added to the phone which can record all conversations/communication and send to a predetermined location without the knowledge of the user!

Its no wonder that people are now abandoning social networks. Many users whom I know are not posting anything on their network and the volume of activity in my social network has dropped by nearly 90% from its peak two years ago. In other words, we have seen the high tide mark of Social Networking.

As users became more aware of the disadvantages of Facebook & Twitter, they switched over to apps such as Snapchat & Whatsapp or LiveChat, etc.  These apps allow discreet communications.

Secrecy has its advantages for people and It's quite attractive. These niche apps were being developed to capture some of few remaining untapped social-media markets.

Now, a  new trend of Apps are coming: The "Anti-Social Apps." While most social networks aim to connect people, one new service seeks to join the growing trend of doing the opposite and help you avoid them.

Some of the new Apps in this category are:

Cloak: 

Cloak 'anti-social' app helps you avoid your friends. Cloak uses location information from other social networks (FourSquare and Instagram at the moment) to work out where your friends are. It then informs you via push notifications so you can avoid anyone you'd prefer not to see. Users can choose to receive an alert when certain people are believed to be nearby.  You can also view friends on a map or on a detailed list telling you how close they are, to the nearest mile. It's free and currently available on IoS.

Snapchat:
Snapchat is used to send pictures and videos - which is automatically deleted seconds after they have been viewed.  This is opposite to that of Facebook, which keeps the photos/videos forever and for everyone to see.

Secret: 
Secret broadcasts messages anonymously - are growing in popularity. Using Secret, one can share a confession with your network (in practice, those of your contacts who also have the app), though they won't know it was shared by you. You must have a minimum of three friends before you start sharing (in order to protect your secrets). In theory, secrets that are liked and reshared can spread across the world — though the app's developers insist you won't be identified at any point. Posts are encrypted so that the Secret team doesn't know what you're sharing.

Whisper: 
Whisper, like Secret enables users to share secrets. One can also respond to other Whispers with your own, and there is a private messaging app if you long to live in a riddle wrapped in a mystery inside an enigma. You can browse by topic, so you can quell your conscience by unearthing others who have cheated on the other halves/shoplifted something/failed to give up their seat on the Tube.  It's free and available on IoS.

Closing Thoughts 

Today's young generation are done with sharing and uploading nearly every part of their lives. Instead, intimacy and privacy in the digital age is now what they seemingly seek. Constantly being connected to the world was cool way back when, but this constant connection has created lots of problems and people are fleeing from it.

Nadella's new Strategy for Microsoft

As the new CEO, Satya Nadella has hit the ground running and has taken several big bets as he defined new strategy for Microsoft.

In this year's Build 2014 conference, Nadella made major announcements which indicate the shift in Microsoft strategy. The new strategy can be summed up in three main points:

1. Fix the current mistakes
2. Embrace the mobile world
3. Build for IoT

Fix the current mistakes

According to a recent study:

• Windows 8.x is installed only on 11.3% of PCs. 
• 78% of the enterprise users have no plans to adopt Windows 8.x! 
• Android PCs are likely to get a market share of 8% by 2015.

This spells trouble for Microsoft and must be fixed ASAP. So Nadella announced few changes on Windows 8.1 - including bringing the Start Menu Back to Windows 8 and boot-to-desktop to make Windows 8.1 valuable to enterprise.

Embrace the mobile world

By making Windows RT 8.1 free for mobile devices with screen size below 9 inches, Microsoft is embracing the mobile world in a new way. With Microsoft's share in Mobile platform at abysmal levels, Nadella had to give away Windows RT for free to attract new cell phone vendors. Microsoft-Nokia has an abysmal market share of 3.1% in smart phones and at this levels Windows OS for mobile will be unsustainable.

So  Nadella is working out a broad strategy to make Microsoft Windows RT more popular by giving it free to cell phone vendors hoping that Cell phone vendors in Asia may embrace Windows more openly. Compared to Android, a free Windows OS would be more attractive - as cell phone vendors need not worry about Patent lawsuits with Apple or get entangled in licensing issues.

Build for IoT (Internet of Things)

Universal Windows Apps. Allow other developers to write apps that works seamlessly on phones, tablets, PC and even Xbox/TV. This is a very significant move from a strategy perspective. This allows Microsoft to leverage its large developer community to develop universal apps and make Microsoft a powerful player in the mobile world.

Microsoft just announced the concept of Universal Windows Apps, the implications are clear. Developers can create Windows apps for IoT - such as Car navigation/control/entertainment system and have the same app be remotely controlled/monitored or augmented in a mobile or a PC and Microsoft Azure Cloud.

If Microsoft enables app developers to leverage all four platforms: IoT, Mobile, PC and Cloud with Universal Windows Apps, then Microsoft will have a major foot hold in the IoT segment.

Closing Thoughts

Nadella has announced a big shift in Microsoft Strategy. Microsoft is fighting with its back against the wall in Mobile space and is non-player in IoT segment. So Microsoft has a stiff uphill battle to make itself relevant in the new mobile & IoT world.

Microsoft is still a very strong player in enterprise software and cloud services. Nadella must leverage these strengths to win the mobile & IoT battle.

Also see: 

Microsoft Azure Re-Branding : Is Microsoft Moving beyond Windows?

Move Over Windows, Android is the New Emperor

Product Management: Leading with Customer Experience

Recently, I was browsing www.flipkart.com and www.indianroots.in and had picked up few items in the online shopping cart, only to abandon it, i.e., I did not complete the buying process - because of slow site response and incomplete product information.

In e-commerce world, its a common problem. An abandoned cart is an online "shopping cart" where a shopper has begun the process to purchase items by adding one or more item to their cart but then leaves the site without completing the purchase.

According to an IBM survey, every year roughly US $83 Billion of sales is lost due to poor customer experience. That's more than the total revenue of all retail e-commerce.

Solving this problem is of paramount importance to any e-retailer. So to begin with, product management must have a seat at the business strategy table and "own" the total customer experience and must work towards improving total customer experience.

In this context, product managers have to work on building better customer experience and improve the customer experience, increase customer engagement to stem such losses from abandoned cart.

Note: In this article, I have taken e-commerce as an example. However the concepts explained here is equally applicable to enterprise software products.

A fresh look at the customer experience– and why it matters

To start with, product management should "own" the customer experience and understand customers. Understanding customer in e-retail implies investing in technology and data analytics and build consumer behavior models and then use this insight to deliver differentiating customer experiences - i.e., tailor the product to match customer needs.

Technology is just an enabler and must not be mistaken for the solution or for strategy. Technology is used to understand the customer and tailor the entire marketing channel and web interactions.

Let me explain this in detail in the next section. Essentially, Technology is used everywhere, but the overall solution depends on strategy - which can be grouped into three distinct steps:

Step-1: Understand Customer Context
Step-2: Act Pro-Actively on the insights
Step-3: Take a broader view of Customer Experiences

Understand Customer Context

Understanding Customers in e-commerce world is a lot more than understanding customer demographics: Age, Sex, Income Group, Geographic Location, User history, user platform (PC, tablet, mobile), Time of browsing/purchase. It also involves knowing which marketing channels the customer has been exposed to or used in the past - i.e, knowing did the customer use the information in a email or social network or on-line advertisement, or search engine.

Knowing all this information that makes up the customer context in real time leads to the next step. Note that the key word here is "real time".

However understanding customer context in real-time is a major challenge. Capturing and understanding the full context of each and every customer interaction across all channels is a gargantuan task - and this calls for high investments in technology.

In the world of e-retail, every single interaction as an opportunity to understand customer context. To do this successfully, e-retailer must be hypersensitive to not only behavior, history and preferences, but also the real-time circumstances customers are facing.

To explain this in simpler terms, think of a old physical retail shop. The customer is greeted by a sales associate who scans the customer demeanor and quickly gauges customer's interests. He can gain additional insight by asking probing questions such as "Why do you need it? Or Where do you plan to use it? And other questions. In e-commerce world, we need to use technology to gain such insight and then tailor the buying experience.

Product management can tackle this issue by automating all processes across channels to calibrate the web responses based on the behavior and context of each individual engagement, which is described in the next section.

Act Proactively on the Insights

Once the customer context is understood, the next step is to act proactively on it. In e-commerce world, this could mean:

1. Adjusting the web site in real-time to match the context and creating right-time, right-place offers.
2. Tracking customer interest and communicating that through different channels in real time and also off-line.
3. Providing a high quality of online customer engagement during the moment customer is on your website.
4. Identifying & rectifying customer struggles (such as slow responses etc.) in real time.
5. Automatically creating customer communication portfolios for the future

All these actions are based on insights gained. This could be due to real time analytics that can serve as  valuable guideposts for creating high quality customer engagement.

Monitoring, tracking and understanding the quantitative value of customer experiences also provides critical insight that can directly influence marketing actions.

Leading e-retailers devote invest heavily to understanding what's going on and then use the resulting knowledge to drive action. They measure results, identify what's working and what isn't, and follow up to verify that their efforts are making a difference.

Real-time action is very important and must be coordinated across all channels of customer interactions: Online, Mobile devices, social media etc., to create an integrated and consistent customer experience.

When real-time understanding of customer context is achieved and used to coordinate activities, it becomes possible to create highly sought-after right-time, right-place offers.

Monitoring, tracking and understanding the quantitative value of customer experiences also provides critical insight that can directly influence the company's actions – and help to ensure that the correct choices are consistently made.

To win customers' mindshare, trust and loyalty, it's essential to coordinate activities and messages across all channels at all times and keep them aligned to the customer context. This  is a highly dynamic and challenging undertaking. But for the customer, it is a seamless experience that sends the message that the company truly knows and understands them.

Take a broader view of Customer Experiences

Each customer experience is an event in itself. Each event taken individually does not constitute an overall customer experience or customer relationship.

Taking a broader view implies in-depth understanding  of the full scope and nature of the engagement with customers and using analytics to guide the future actions and build customer relationships that grow and strengthen over time.

Customer experience is influenced by every customer touch point, even the call center, website, email, mobile app and shipping.  Customer experience can be improved by addressing any difficulty tied to that experience. Note that every single customer touch point creates a customer experience, from the website, Mobile app, Facebook business page, Email marketing campaign etc. All the details from each of the customer touch points create an overall customer experience.

All leading e-commerce companies maintain all transactional history which is then used in future. Taking a broader view of customer experience helps in enhancing the lifetime value of the customer. Leading e-commerce companies integrate all inbound actions with outbound communications across multiple channels. For example, if customer had abandoned a cart, a follow up email is sent requesting customer to complete the transaction, while the online advertisement points to the product that was in that abandoned cart.

Product Management, having taken up ownership of customer experience and sets up broad strategic goals such as:

1. Complete channel integration.
All channel activities should be mapped and prioritized to foster integration for creating a unified customer communications and touch points.

2. Building an insight-driven organization.
Technology must be used to create a holistic view of the customer relationship and this insight be understood by the entire organization and act on relevant insights.

3. Build an enterprise wide focus on customer experience.
It is essential to understand in depth how customers engage with your website to identify opportunities to serve them better. This implies placing the customer at the center of the business.

While these are long term goals and has a direct impact on the financial front. But these goals also have important challenges. Integration issues will continue to exist as the number of channels and the channels characteristics change with new technology. Even today, leading e-retaileres struggle integrating mobile and social channels with other campaigns and tactics.

Product Management should also have a clear vision of the end goals for each quarter/year. As these goals are tough to achieve, it needs to be broken down into smaller end goals for each quarter. From a product planning perspective, building superior customer experience is a journey with several milestones. Product management lays out the plan for the enterprise and the shows the way forward.

Putting Customer Experiences First

Enhancing customer experience should be part of the overall strategy. Product management defines this strategy and gets a buy-in from all stake holders. Once the strategy is agreed upon, product management has to break down customer experience into measurable metrics, identifies steps to achieve those goals and then give directions to each departments/groups to reach those goals.

In the initial stage, the goal could be to create technologies to measure those metrics.
The broad strategy can be like:
1. How can the company orchestrate the engagement of its customers across all its channels?
2. How to understand the context and motivations of each customer engagement?
3. How to differentiate and create highly personalized customer engagement?
4. How to create new and unique communications?

Leading e-retailers across the world are investing to find the answers to these questions and then set about creating superior multi-channel experiences by making full use of every tool available to them.

E-retailers should go beyond each transaction and follow up with a definite action plan to create a very positive user experience. For example:
1. Contacting customers to get their feedback on their experience.
2. Monitoring/Tracking delivery commitments to ensure customer fulfillment
3. Identifying cross-sell and up-sell opportunities.
4. Designing other customized offers for future use

Consider how Amazon has taken ownership of the entire customer experience by using technology, business analytics,  insights and engagement in a carefully orchestrated way. When customers log into Amazon website, they encounter a seamless blend of live web information and interaction, relevant web-store displays that is both customized to them and made more convenient by presenting it in multiple platforms.

The entire process flows smoothly across channels. For example if a customer has clicked on a web advertisement and he is instantly taken to the relevant product page. The page also shows all other products that are relevant to the customer.

This sends a powerful message that Amazon has invested in building a better customer relationship by making it easier to interact and conduct transactions. This cross-channel integration displays broad understanding of the customer, knowledge of customer context and ability to take systematic action, all brought to life through a single customer interaction.

Facebook To Buy Oculus VR for $2 Billion!

Today, March 26th 2014, Facebook has announced plans to purchase Oculus VR, the company behind the Rift headset, for around $2 billion in cash and stock. This includes $400 million, and 23.1 million Facebook shares.

This acquisition indicates the value of Internet of Things (IoT) and brings new platform for Social Networking, taking users of Social Networking to a new level of Virtual Reality interactions.

Oculus VR brings in multiple technologies to force. First is a wearable mobile device Oculus Rift headset, a connected device (Like Google Glass an IoT), but its a lot more involved, the Virtual Reality technology will completely immerse the users into a virtual world.

When this technology is merged with Social Networking, one can create new experiences - which was impossible before. For example, a group of friends can play a game of Soccer - Virtually!

People can meet virtually and yet have a real life like experience. Oculus makes a virtual reality technology, the Oculus Rift headset, that allows you to enter a completely immersive computer-generated environment, like a game or a movie scene or a place far away. This incredible technology makes you feel like you are actually present in another place with other people. Its unlike anything one has experience before.

Actually, this level of Virtual Reality (VR) has been used in expensive flight simulators and other military training equipment. Oculus brings that platform into a more affordable price for general public.

"I'm excited to announce that we've agreed to acquire Oculus VR, the leader in virtual reality technology," said Facebook CEO Mark Zuckerberg in a statement today.

Huge Impact on Facebook

Facebook is no tech slouch. Facebook has tremendous investments in new technologies - such as is face recognition technology called "DeepFace Program". It is also creating a host of new programming tools such as Facebook Hack.

Recently Facebook acquired WhatsApp, which brings in real-time mobile to mobile communication technology. And now with Oculus acquisition, Facebook can take this VR technology, mobile-to-mobile communication technology from WhatsApp and merge both of it to its social networking platform & with new programming tools & visual processing technology,  Facebook can create a new experiences for its users. And hope that youth of the world will continue to use Facebook, and prevent the decline in its market base.

"Over the next 10 years, virtual reality will become ubiquitous, affordable, and transformative," says an Oculus blog post. And with Oculus VR platform Facebook can change social networking forever.

Also see: 

Next Technology Disruptor: Internet of Things

Why is Facebook buying WhatsApp for $19Billion?

How to identify and create disruptive innovation that customer would buy?

Microsoft Azure Re-Branding : Is Microsoft Moving beyond Windows?

In my last article, I had written than 2014 will the year, Windows will lose the OS crown and Google's Android will be the new emperor.

Today, I got to know that Microsoft has started acknowledging this change and is re-branding its Windows Azure Cloud Service as Microsoft Azure.

The re-branding exercise is a BIG message. Satya Nadella, the new CEO of Microsoft is sending a message to the whole world that Microsoft is moving beyond Windows and he will transform the company in new ways.

Azure Cloud Services

For a long time, Microsoft's Windows Azure Cloud services was the second largest cloud offering and today Azure is a cloud power to reckon with. During this time, cloud services has grown and matured and is now ready for prime time. In face of this change, Azure must adopt and embrace an Open & Extensible cloud infrastructure. Else customers will move to other service providers such as RackSpace, Vmware's vCHS, or Verizon's Terremark, etc.

As a product manager at EMC's ASD division which makes solutions for building private/public cloud services,   I have always preached the need for an open and extensible system and to me, it looks like Microsoft is finally embracing a Open and Extensible cloud services, and re-branding exercise is a very public acknowledgment.

Microsoft Azure - An Open Cloud

For a long time, Azure, ever since its creation supported only Windows OS to be run on the virtual machines, and also restricted it to Microsoft tools - MS-SQL, .Net, etc.

But now things have changed and once run Oracle databases & middle ware, Java, PHP, Phyton etc. Today, customers can even run Linux as a guest OS on the virtual machines.

As Microsoft embraces Open architecture, I am sure, Microsoft will allow customers to use VMware ESX hypervisor and Linux or Android or Windows as guest OS. Microsoft will open up Azure to a host of other non-Microsoft technologies as well.

Microsoft Azure - An Extensible Cloud

Azure was always a public cloud, run on Microsoft premises and managed by Microsoft. Now with the launch of Vmware's vCHS - which is a hybrid cloud, Microsoft will have follow suit and allow hybrid cloud services as well - where customers can choose to use Azure in their private cloud or public cloud or a mix of both. This implies that Azure platform will have to be made open and extensible to interact with customer's private cloud.

I would predict that Microsoft will soon release a new set of cloud development tools to help customers extend Azure cloud services to their in-house private clouds and also enable customers to move data & compute workloads in & out of Azure's public cloud.

Closing Thoughts

As Windows loses its relevance in the personal computing space. Microsoft is extending its Office suite to iPad. Microsoft is expected to launch Office of iPad this Thursday March 27 2014.

All this re-positioning indicates that Satya Nadella is determined to change Microsoft and move the company beyond the Windows world!

Also see: 

Move Over Windows, Android is the New Emperor

Move Over Windows, Android is the New Emperor

Year 2014 marks not just a change of leadership at Microsoft, with Satya Nadella taking over helm from Steve Ballmer, but it is also the year when Microsoft Windows steps down and hands over the crown to Android.

About an year ago in March 2013, I made a bold prediction that Android PCs will become mainstream. Now in March 2014, all the leading PC vendors are releasing Android based PCs.

Its been a well known fact that PC sales had been declining for last 5 years. And by end of 2014, Windows has completely lost its position in Personal Computing space. ( Though Windows will continue to be relevant OS on the servers)

Android OS is already ruling the end user compute market with smart phones and tablets (with a market share of over 75%), and in 2014, both Lenevo, Acer & HP will release PCs with Android pre-installed.

Sales of desktop and notebook systems are plummeting as consumers choose to spend their money on smartphones and tablets, so PC OEMs have to come up with ways to make PCs sexy again. And one idea that some OEMs are trying is replacing Windows with ... Android.

Acer TA272 HUL, running Android 4.2 Jelly Bean, has a 27" screen with resolution of 2560 by 1440 pixels, has 16GB of storage, and also supports a key board and mouse - although it has a full touch screen.

While there are still few drawbacks in Android OS, the major one being multi-tasking and does not support Adobe Flash, it is still a complete home computer and is ideal for home use.

Why is Windows Dead

Windows 8 is far the best OS for the PC. When it comes to dealing with work loads such as creating audio & video content, graphics, and multi-tasking, Windows 8 is far superior to Android, but most PC users do not create content on their PCs, at the most, they may write a few documents, edit pictures or videos and write emails. The most popular uses of PC for most users are web browsing, social networking and entertainment: gaming or movies. For these applications, Android and Google Apps are perfect. For example, I can do all my home computing work on a Android MiniPC, and best of all, all my work is saved automatically on web. So I don't have to worry about backing it up or carrying hard drives around.

But the real killer of Windows PC is not its capability to deal with traditional work loads. The Windows killer are the completely new capabilities which comes with Internet of things. New compute devices such as Smart Lamps, Car management systems, wearable devices etc., are being built on Android platform and these smart devices do not have connectivity to the windows based PC!

The seamless integration of these new smart devices with Android platform - makes Android PC a very compelling argument to replace the old traditional Windows PC.

Dominance of Android

Android already dominates two of the three platforms:

1. Android already rules the Mobile & Tablet Space - where it has 75% market share
2. Internet-of-things is being built on Android
3. Android can unify everything by becoming the OS of choice in the PC platform.

As I write this article, several CIO's in various organizations are looking at replacing their PCs with virtual desktops (VDI), and I see eight good reasons to opt for Android based PC for deploying Virtual Desktops.

1. Price
With minimal hardware requirement and free software, the cost of basic hardware is so low that Windows based PC's cannot match.

2. Seemless usage
Android has seamless integration with Google Apps. For most enterprise office requirements, Google Apps is adequate. For other apps, users can subscribe to s SaaS service - such as Office365, Saleforce.com, and other SaaS services.

3. Transparent control
IT departments can easily control the SaaS and Virtual desktops. Power users can use the virtual desktops that are running on powerful servers and the central IT department can roll out just the Virtual Desktop service, which can be accessed either on any Android platform. Another major advantage here is that all data is in the cloud - either private or public, this makes it easier it to secure and check for compliance.

4. Cost management
For enterprise IT groups, supporting different PC systems: Laptops, Desktops, Workstations etc., is a major cost and operational headache. Now with Android and BYOD (Bring Your Own Device), enterprises can bring in efficiency of ITaaS (IT as a Service), and have better control over the costs through VDI.

5. Performance
Android is a very simple platform and does not demand high compute resources. Power users can be directed to the right types of VDI, thus eliminating the need for multiple hardware platforms. A simple 1GHz dual core ARM CPU with 4GB RAM gives just the same level performance as a top end x86 laptops. And with a boot time in seconds, users will love the performance.

6. Browser based Workloads
With cloud computing becoming prevalent, all the work is done through a web browser -- so with Android OS well integrated to the best browser, users will have best possible user experience.

7: Apps and add-ons
Android platform has millions of apps, and a very good Mobile device Management (MDM) tools which can control the apps users can access, Users will have all the right tools they need for the future. As Internet of things becomes popular, users will find that Android based apps will allow them to seamlessly integrate with the new devices and new types of business focused tools can be developed.

8: Elegance
With a full touch screen and ultra high resolution screens in sleek formfactors, users will love their Android PCs. It will rival the MacBook Pros and MacBook Air laptops. The looks coupled with high performance and efficiency will make Android PCs the ultimate end-user computing device.

Closing Thoughts 

The failure of Windows 8.x and the confusion Microsoft created with its Surface tablet, Microsoft has annoyed all its partners and is running short on friends. PC vendors are forced to look for new market opportunities.

In 2014, Considering its historically install base, Windows will still be the top OS for PCs. But by 2016, Windows will be relegated to history books.

Also see: 

Android PC - Is just around the Corner
PRODUCT REVIEW – ANDROID 4.0 MINI PC

Understanding Data in Big Data

Data forms the foundation for all Big Data Analytics. The first step in Big Data Project is to define the business objective. The second step is to understand the data needed for that big data project.

To make the most of big data, you have to start with the data you understand and trust.

To understand this better, lets continue on our retail example.

In many retail operations, there are multiple ID's for customer. There is a customer ID, Account number, Loyalty Card Number, Customer Email ID, Customer Web Login ID. All these Ids were is different data types & formats.

So when the company started a big data project, defining the term customer ID was a big challenge. Adding to this confusion, each department has a different ways/systems to track customer purchases. This meant that project team had to spend extra time and efforts to identify and document various data sources and determine which data to use in which reports. While business leaders were hampered with multiple & inconsistent reports coming from duplicated or missed data.

To harness the power of big data, companies must ensure that source of all information is trustworthy and protected.  We call this as Master Data - which represents the absolute truth and forms the basis of all analytics. Once the master data is created & protected, organizations can trust the analytics and take decisions.

It is equally important to have a common business glossary for key terms and data sets.  Once a common terminology is established, the next step is to define where each piece of data will come from and where it will go - i.e, define all sources of data and the destination of that data. This will help everyone involved in the big data project know exactly what each data terms mean, what each key metrics mean, where the data should originate, and where the data will be stored - thus establishing a "universal truth" for all the business data.

Trustworthy Data

Business leaders are eager to harness the power of big data. However, as the opportunity increases, ensuring that source information is trustworthy and protected becomes exponentially more difficult. If this trustworthiness issue is not addressed directly, end users may lose confidence  in the insights generated from their data— which can result in a failure to act on opportunities or against threats.

To make the most of big data, you have to start with data you trust.

Defining universal truth is just a first step. Next step is to secure that the data being generated from the source is indeed "trustworthy".  Unfortunately, it is not enough just to establish policies and definitions and hope that people will follow them. To have a truly "trustworthy" data, organizations must be able to trace its path through their systems & have adequate security systems to ensure that the data follows the defined paths only and does not get tampered with.

Data collected by the system must be controlled via some form of access and version control, so that the original data is not lost and any changes made can be tracked. There are lots of tools in the industry to secure & track data.

Defining "good data" 

In case of big data, remember that the volume of data can be really large and can easily overwhelm any data collection system. So the data collection system must be tuned to collect only the "Good Data" - i.e., data that is useful in analysis, reporting and decision making.

For example, sensor in a retail shop can collect data on number of people at a particular isle will keep collecting data even when the store is closed. It can also collect lots of other data such as ambient temperature etc., which is may be irrelevant to the analysis.

So organizations must be able to extract only those bits of data necessary to support a particular objective. This way unnecessary data can be kept out of the system and avoid hardware/software costs.

Data Lake

Once all the good data is identified and collected, it has to be pooled together for analysis tools to work. Technically, one need not pool all the data into one system, instead knowing where the good data is stored in a safe trustworthy location - which together forms the data lake - a collection of all valid data.

Please note that Big Data is also characterized by "Variety", "Volume",  & "Velocity".

Continuing on our retail operations example, There are a whole lot of  "Variety" of data sources that can be tracked:

1. RFID data,
2. Sensor/Camera data
3. Data from Social Networking sites
4. Data from Website
5. Data from Twitter or SMS
6. Data from Point-of-Sale devices

Variety of data souks also implies different data types: Structured transactional data, Unstructured video/camera data, and Metadata. So you now have multiple sources of data feeding in huge volumes of data at a rapid rate. This implies that your analytics must be able to process this large volume of data in a reasonable time - which can give analytical results in a meaningful way.

Often times, companies do not want to process all the data in real time - because of business objectives and due to customer behavior. For example, customers may react to a marketing campaign at different times, and this reaction can be seen in social networking sites, or tweets, or SMS, or email or web comments at different times. So one must collect the data over a period of time for any useful analysis. This results in creating a system that can handle this huge volume of data in the data lake.

Data Lake Illustrated

All the data collected could reside in multiple locations - which is logically pooled together to form a "Data Lake".

Data collected in this data lake extends beyond on-premise applications and data warehouses. It may include, data from social networking sites, customer tweets, web sites, etc. This type of external data can be harder to collect and analyze than traditional transactional data. Potential insights can be buried in unstructured documents such as: User generated documents,  spreadsheets, reports, email and multimedia files.

All this data is collected and secured in this data lake. The secure trusted data in the data lake then forms the basis for the big data analytics project.

Data from multiple sources is collected, sometime selectively to limit the volume of data, and even processed to make the unstructured data more useful. Even the metadata - i.e, the data which describes the main data is also collected. For example, in case of a photograph, time/date, location, type of camera, ambient conditions at that time, etc. are all metadata. Metadata is very important in understanding unstructured data.

Data lakes act as repositories of all valid information: Log files, User database, transactional information, behavioral information. Analytics is often run on the data stored in the data lake.

Analyzing big data at rest 

By analyzing large volumes of stored data, organizations can discover patterns and insights that allow them to optimize processes and profitability. In environments where you quickly need to gain insight on hundreds of millions or even billions of records, the ability to match master data at the big-data scale is a must-have. Examples include reconciling large lead lists or matching citizens or customers against watch lists. Additionally, organizations want to analyze large volumes of stored data to discover patterns and insights.

Analyzing big data in motion

The data lake must be capable of handling the high volumes of data that is being generated.

With certain kinds of data - often coming from sensors, there is no time to store it before acting on it because it is constantly changing. In cases of fraud detection, or health care issue or traffic management etc.,. In such cases gaining real time insights with high speed real time analytics is vital. High-velocity, high-volume data calls for in-motion analytics.

Handling such volumes of data can be daunting. How to analyze hundreds of millions of information bits in real time calls for systems that can analyze data in motion. These data in motion analytics require dedicated systems to process data as it gets generated and alerts are sent in real time. In addition the processed data is also captured in the data lake for future analysis.

Data in motion analytics systems analyzes large data volumes with micro-latency. Rather than accumulating and storing data first, the software analyzes data as it flows in and identifies conditions that trigger alerts - such as credit card frauds. Along with the alert, the processed data is also stored in the data lake and can be analyzed with other data for better business outcomes.

Securing Data Lake

The main advantage of creating a data lake is that it acts as a single repository of all data, which users can trust and use in their analytics. Access to this data lake can be controlled and have a unified data protection systems. Data lakes will require role-based access, policies, and policy enforcement.

Often times, all data entering into the data lake are tagged and metadata is added. Metadata information on data security, access controls end user rights are tagged to the data. Each data element is tagged with a set of metadata tags and attributes that describes the data, who can access it, and how it should be accessed and handle it. This tagging is rule based  and can be enforced with data management tools.

Protecting the entire data lake is often cheaper than securing each individual component of data. This way, organization can check, monitor and report on data security, privacy and regulatory compliance. Data lakes thus acts as single secure & trusted repository of information.

This is the third part in the Understanding Big Data series. The first two parts are:

Also see:
Part-1. Understanding big data 
Part-2. Defining the Business Objectives for Big Data Projects

Defining the Business Objectives for Big Data Projects

Business leaders need to clearly understand the Business Objectives before embarking on a Big Data project. The first step is to define the business objectives. In a recent survey by IDC, the most popular business objectives of Big Data projects are:

1. Customer Centric Outcomes
2. Operational Optimization
3. Risk Management
4. Financial Management
5. New Business Model
6. HR Analytics

The objectives have to be well defined from a business perspective, else the bag data project will fail. Remember that with big data, you are searching for a needle in a hay stack - but to find that needle, you need to know what the needle looks like, behaves like and feels like.

Every business objective of any big data project  has four main parts:

1. Context of the project
2. Needs that the project is trying to meet
3. Vision of what success might look like
4. Outcome in terms of how the organization will benefit from the result

All Big Data Projects must start by defining the objectives. It could take anywhere from a few hours with one person to months or years with a large team. Even the briefest of projects will benefit from some time spent thinking up front.

Now, lets take a deeper look at these four main parts of business objectives:

Context

Every business objective has a context. It is the defining problem that the big data project must solve, Who are the people interested in the results, what are they trying to achieve?

Contexts emerge from understanding who are interested, say for example Sales managers, or Marketing managers or Finance Managers etc., and why they are doing what they are doing?

The context sets the overall tone for the business objectives and guides the big data project. Context provides the background to the project and help define the mission. The Context provides a project with larger goals and helps to keep us on track. Contexts include larger relevant details, like deadlines, that will help us to prioritize our work.

Context comes from people, so with time new contexts emerge as new groups of  people come onboard and have different missions.

Here are few examples of contexts:

• A marketing manager wants to know the effectiveness of the marketing in social media.
• A sales manager wants to predict the how much he will sell next week 
• A store manager wants to know how often customers visit a particular store.  

Needs

Correctly identifying needs is critical. By clearly explaining the needs, business objectives of Big Data projects can be well defined, planned and accomplished.  The need must be defined, perhaps a definite problem has to be well articulated.

When we correctly explain a need, we are clearly laying out what it is that could be improved by better knowledge that can be gained by the Big Data project.

Business faces challenges. And these challenges results in specific needs, That that could be fixed by intelligent data analysis. These needs should be presented in terms that are meaningful to the organization. If your method will be to build a model, the need is not to build a model. The need is to solve the problem that having the model will solve.

Big Data analytics is the application of math and computers to solve problems. Business leaders will have to determine which questions can be answered by data analysis.

Continuing on our retail example, Here are some fairly common needs:

Business leader wants to expand operations to a new location. Which one is likely to be most profitable?
Some of our customers leave our website too quickly. We don't understand who they are, where they are from, or when they leave, and we need to know how to retain them.
As a business leader, I need to choose between two marketing campaigns and choose the most effective one.

Vision

Vision is a glimpse of what the results of business objectives will look like - even before implementing a big data project. It is like a mockup, describing the intended results.

Think of vision as a model of a house - a mockup without much low level details - but just enough to give a good picture of what the finished house will look like.

A mockup is a low-detail idealization of what the final result of all the work might look like. Keep in mind that a mockup is not the actual answer we expect to arrive at. Instead, a mockup is an example of the kind of result we would expect, an illustration of the form that results might take. Whether we are designing a tool or pulling data together, concrete knowledge of what we are aiming at is incredibly valuable. Mockups, in one form or another, are the single most useful tool for creating focused, useful big data analytics to work.

Having a good vision is the part of scoping that is most dependent on experience. The ideas we will be able to come up with will mostly be variations on things that we have seen before. There is no shortcut to gaining experience, but there is a fast way to learn from your mistakes, and with big data analytics, you can try out as many of them as you can. Especially if you are just getting started and work on a smaller data set, you can visualize outcomes even before confirming it with complete analytics.

Vision can take the form of a few sentences reporting the outcome of an analysis. For example a simplified graph that illustrates a relationship between variables, or a user interface sketch that captures how people might use a tool.  The Vision primes our imagination and starts the wheels turning about what we need to assemble to meet the need.

Before you start a big data project, you need some vision of where we are going and what it might look like to achieve your goal.

Continuing again with our retail example:

Need: A retailer is trying to measure its successes of its email marketing campaign.
Vision: Present key performance indicators on a regular basis. It will consist of graphs and text.

Need: A retailer is looking for new locations to expand.
Vision: Report that shows each location and expected sales for each location.

The most useful part of creating mockups is that it lets you work backward to fill in the data of what we are actually looking for. If you are looking key performance indicators of an email campaign, then you know where to look for information and come up with metrics and valuation models. This help to get all pieces fall into place faster.

Outcome

Lets assume that we have all the results, then we need to understand how this insight will help the organization and what will happen once the reports and insights are generated.

How will it be used?
How will it be integrated into the organization?
Who will own this reports/Actions?
Who will use it?
How will its success be measured?

If you don't understand the intended use of the reports that is being produced, then it will easily get lost on the corporate jungle and all the hard work will go to waste.

The purpose of the big data project must be established while defining the business objectives.

Figuring out what the right outcomes are boils down to three things.

First: Who will see the results? List of people who will use the reports/insights. The persons who will see & use the reports/insights must have the skills to interpret the results, give valuable feedback to the project team and explain any modifications to the initial business objectives (if any).

Second : Who will maintain this big data analytics system. Is this a one time activity or does the organization need repeated runs of this analytics? If there is a need for continuous runs, then who will own and maintain the system?
Who are they? What are their requirements?
What should the project do to build a repeatable & sustainable big data analytics?

Third: What will be the business outcome of this analytics? What will change with this reports? How can we verify that concerned leaders are taking  suitable action based on the analytic reports?

It is very important to think through the outcomes before embarking on a project. Therefore the outcomes must be defined into the business objectives.

A good and well defined business objective is the one which has the defined the context, Identified the right needs, and has a good vision of what the results might look like and gives the required outcomes - so that the big data analytic project reports will do something useful.

Seeing the Big Picture

Successful Big Data Analytics Projects start with a well defined business objectives. The business objective needs have a coherent narrative about what the organization might accomplish by analyzing big data, and what problem it is solving.

Once the business objectives are defined, the next step is to identify the right tools and methods to use. Just focusing on maths or software - without a a good business objective will result in wasted time and energy.

Also see:

Understanding Big Data

Understanding Big Data

Recently, I was invited to give a talk  to engineering college students about big data, and I decided to document it & here it is.

Big Data is now a hot buzz word in IT sector. Business leaders are eager to harness the power of big data and are spending serious amounts on money on it.

With Big data, companies can predict next months' sales, can predict future pricing, better understand customer preferences in ways unimaginable before.

While big data promises the sky with a beautiful rainbow, it is really a big challenge to successfully harness the power of big data. To begin with, lets understand what is big data.

Big data is just that - huge volumes of data which the current IT systems cannot handle and organizations do not process this data as this data does not fit into the current databases or data warehousing technologies.

To understand this better, let me explain with an example.

Consider a retail operation and you would like to know: How did sales this month compare to the previous month and also compare with the same month last year?

The traditional way of reporting this will be look at sales transactions & look at sales revenues, also look at ERP systems and report on the merchandise sold. But this does not give the full picture of customers. It does not capture the number of footfalls. It does not tell the number of items customers looked at before buying one product. It does not tell how much time each customer spent in the shop, it does not tell what items customer wanted to buy - but could not do so due to lack of inventory in that shop. So in other words, the traditional IT reporting systems present only a partial - a tiny bit of the full picture.

While Big Data can do this by capturing data from various other systems.

What is Big Data?

Big data is characterized by four parameters: Variety, Volume, Velocity and Veracity. These are often referred to as 4V's.

Variety

Big Data in simple terms implies looking at the same parameters, in our example of monthly sales, in different ways - which is often called as Variety of Data Sources

Continuing on our retail operations example, There are a whole lot of  "Variety" of data sources that can be tracked:

1. RFID enabled shopping carts to tell how much time each customer spent in the shop.
2. RFID readers in the cart to identify which items customer put in the cart and then did not buy.
3. Cameras and sensors that shows which areas of the shop was overcrowded and that prevented customers from shopping in that isle.
4. Sensors & RFID readers that show which items customers looked at before selecting an item.
5. Sensors data which tells when each customer came to the shop
6. Sensor data which tells the time taken at the billing section.
7. Internet search information from each customer at the online store.

This type of wide variety of data is valuable, yet today business IT systems cannot handle it or even use it for analyzing monthly sales.

Wide variety of data sources also implies that IT systems must be agile to accommodate a wide variety of data types: Transaction data, Video feeds, sensor data etc and seamlessly integrate with various data processing technologies.

All of this data is stored in a dizzying array of formats. Some gets stored in structured formats in databases or enterprise resource planning (ERP) applications, but much of the video feeds, sensor data, photos are all unstructured data formats that must be managed, stored and processed.

Data doesn't rest once it is in storage. It must be moved from application to application and from system to system so managers and executives can interpret the data and come to meaningful conclusions.

Volume

In traditional IT systems, each customer transaction translated to few KB of data, but if we were to consider all the different variety of data sources, the volume of data generated per transaction will run in several Mega Bytes or even Giga bytes!

Increasing data volume is at the heart of the big data challenge. Large data volumes can cause many obvious technical problems,  such as excessive batch processing times, bottlenecks and so on.

The sheer volume and complexity of big data means that the traditional method of managing data does not apply to this new sources of data. A complete new system has to be built to collect, governm and process this large volumes of data. New automation systems are needed for integration, governance and using it.

Researchers at  IDC estimate that by the end of 2013,  the amount of stored data will exceed  4 zettabytes, or 4 billion terabytes. That's  50 percent more data than the digital universe held at the end of 2012, and  four times as much as in 2010.

Velocity

Every second of every day, businesses generate more data. What used to be several MB of data per hour with the older systems, is now several GB!

In our retail example, data is coming in several mega bytes every second. Data is coming in faster than ever. Data from video cameras, RFID readers, sensors are being collected in seconds and microseconds.

The IT systems must be able to understand data as it is streaming in, store that information, quickly process this information, and move data quickly from one application or repository to another,  where it can be processed and analyzed.

Unfortunately, many of the older data integration solutions lack the high performance that  big data projects require. There is not enough time to collect the data and process it in real time or near real-time.

Validity or Veracity

The first three Vs: Variety, Volume and Velocity define big data.

But the fourth V veracity is most important for business analytics.

Veracity is the validity of data. Once the data is collected and stored, understanding which of the data sets is valid for a particular business analysis is of paramount importance. Running analysis on invalid data results in invalid or useless results.

For Big data to be valuable, the data must be valid.

Often, big data is collected into a vast pool of data - which is often referred as "Data Lake". Big data coming into this data lake has be indexed, categorized and verified to ensure that the data is accurate, current and complete - before running analysis and making business decisions.

For any organization to take advantage of the opportunities available with big data, it must have IT systems, processes and solutions that can handle all four Vs.

It starts with discovering different sources of data, setting up systems for collecting that data, building  IT systems to govern, process and store large volumes of data. The IT system must be agile and capable to accommodate a wide variety of data and seamlessly integrate with various technologies and it must automatically discover, protect and monitor sensitive information as part of big data applications.

Big data, big opportunities

Big data presents big opportunities for increased growth and profitability.

Organization recognize that big data contains valuable information. They are eager to analyze it to obtain actionable insights that could help them take better decisions that helps improve sales, profits and identify new revenue opportunities.

Forward thinking organizations - such as Flipkart & Myntra in India, Amazon, Wal-Mart, WholeFoods in the USA are already realizing some of these benefits.

• Myntra plans to get revenues of Rs 1,500 Crores by 2015.
• Flipkart is expected to have a revenue of Rs 6000 Crores by 2014.
• WholeFoods - a specilty food retailer uses social media data to analyze customer sentiment  and improve loyalty, helping to drive revenue growth.
• Fedex uses machine data to optimize logistics, thereby reducing shipping costs.

While Big data promises big advantages, but capturing these sorts of benefits from big data requires knowing what the business needs and being able to find key items within the larger mass of big data.
One has to start with articulating the business goals, and that helps determine:

1. What data to collect
2. When & how to collect
3. How to store & process

Then only business can generate the analytics and reports necessary to support those objectives. Now business leaders can make meaningful decisions.  

Next Technology Disruptor: Internet of Things

2014 is the year when Internet of Things becomes are reality & it will start disrupting the tech industry

2014 is shaping up to be interesting year in the tech industry. Several new & exciting technologies are being introduced. The main focus this year is to move mobile smart devices beyond the cell phone space - and create exciting new products:

1. Smart Watches:  Samsung Gear & Gear-2 NEO. 
2. Smart bands: Nike Smart Band, Hauwei Talkband, GoQii
3. Smart Glasses: Google Glass, Vuzix M100, Epson Moverio BT-200 
4. Android in Cars: Open Automotive Alliance, OpenXC 
5. Home Monitoring: Nest Thermostat.
6. Subscriber Identity Module: Nymi acts as a digital vault and password holder.
7. Industry Appliances

The term "Internet of things" emerged as a buzzword over the last year or so to describe the phenomenon of network-connected devices, such as thermostats, that in the past were simple standalone appliances.

The phrase "Internet of things" stands for devices that can be connected wirelessly to Internet and other compute devices - such as phones or tablets or directly to Internet Cloud.

These new connected devices is a gold mine for Telecom companies, car manufacturers, software/App vendors and networking equipment makers. But this new technology can also disrupt their existing business.

The writing on the wall for tech companies is clear: The Internet of things is coming, and you better disrupt or prepare to be disrupted.

The number of these smart & connected devices will grow rapidly and is expected to reach 500 billion or more by the end of the decade. When we think about such large number of connected devices generating huge volumes of data - it will change the very nature of technology industry.

The financial rewards of this new technology is huge. It will touch every aspect of our lives and disrupt every single industry we know of - just like Internet did 25 years ago.

To understand how this Internet of Things can change things, consider the game of basketball. Sensors on the player's shoes can tell how well the players are performing and help coaches plan a better game - by using real time analysis of player & ball movements.

GoQii can change how professional athletes train. GoQii can capture all the movements, exercise, heart rates, calories burnt, sleep patterns etc and feed that information to a real time analysis systems that can help players & coaches plan better.

The possibilities to change things are endless. Successful companies will have to built easy-to-use, high value apps and that will happen over time.

And there are dangers as well. Loss of privacy, digital identity theft and ever widening of the digital divide are the biggest risks. Legal systems must evolve to deal with this new everything-connected world.

Companies around the world are still grappling on how to take advantage of this new technology and many will perish, while several new ones will emerge.

Closing Thoughts

The benefits are so massive that it is inevitable that Internet of Things will be everywhere. By connecting devices over the Internet/Wireless over mobile networks, companies can manage a wide range of new services for their customers.

Leading companies are not waiting for market to emerge. Google paid $3 Billion for Nest's Smart Thermostat technology! Soon others will follow with big money and explore new opportunities.

As the devices get smaller and smarter, as well as the circuitry that drives them, we will have more products that integrate into our life. Soon the question will not be: Whether you're wearing a piece of smart tech, but why not?

Internet of Things is now the new gold rush for this decade!

Also see:

Mobile+Cloud is the Next Big Thing

Mobile+Cloud is the Next Big Thing

Last week, I bought a Lenevo Smartphone for my mother. It costed less than $100 (In India, we pay the full price for the handsets and there are no carriers subsidies)

The new phone comes with 16GB memory, 1 GB RAM, HD display, Andriod 4.3 Jellybean 16GB Flash, 5MP Camera, etc. In short the new phone is saturated with features and can do pretty much everything one can do with a high end iPhone.

Today, Smartphones and tablets are powerful enough to run applications that was once found only on a PC. The Network & Internet is everywhere. With 4G, Smartphones can download faster than most DSL services.

As hardware capability on a mobile platform matures to point where the next big innovation will happen in software. Some of this software innovation will be on the device - but big part of this innovation will be in the cloud.

Service providers and Startups and innovators are looking at means to make smartphones more useful and more valuable to customers. The revenue/customer from traditional phone services: Calls, SMS, & Data is declining and the value is shifting to the Apps. Dropbox is currently valued at $10Billion, Facebook spent $19B for WhatsApp! This acquisition is primarily driven by the potential value in the Apps - and not in the cellphone service.

Going by valuation alone, the total value of App companies is more than the combined value of phone companies!

As this shift happens, the use of cloud by mobile Apps will accelerate and create a new wave of innovation.

Types of Mobile Apps

As mobile phones evolved, mobile apps have also evolved and the apps can be classified broadly into 3 categories:

1. Generation-1 App: Basic Phone Apps
Application uses only the phone resources. These applications are tightly tied to the phone. Eg: iWork suite.

2. Generation-2 App: Web Applications. 
The Phone apps is a rehashed version of the web application. Eg: Facebook, Gmail, etc. The same application can be accessed on a PC or on a smartphone.

3. Generation-3 App: Cloud Apps. 
Cloud Apps leverage the compute & storage capabilities of the cloud and little phone resources to deliver unique experiences to users. For eg: Dropbox, Snapchat, Mobile gaming, etc.

Today, Mobile devices like smartphones and tablets are becoming devices to access cloud resources. As the trend catches on, the use of cloud by mobile applications will be the leading use of cloud computing, surpassing personal computers. And as that happens development of new and innovative applications - which will enable users to exploit cloud computing in multiple ways.

This shift is happening as we speak. Phone service providers and transforming into cloud service providers. Some carriers are moving the base station functionality from the cell site to a cloud data center. These cloud datacenters allow phone service providers with tremendous compute capability and that capability can now be used by new Gen-3 Apps.

For example, how about getting real time traffic & weather information on the route you are travelling on your mobile. Say for example alerting you of a traffic jam 400 meters ahead!
Another example could be relaying information about the best deals in a shopping mall to users based on user preference etc.

Phone Service providers and Cloud service providers can offer new Apps - that can give real time information to users. Cloud computing capability provides huge number crunching capability to process Big data, and end users can consume this results from their mobile phones - in new and unique ways.
For example, creating a mashup of Airline travel information and weather information can help a local taxi driver get more business.

Mobile network and phones will be rapidly commoditized in the cloud era, Forcing service providers to innovate and become platform providers or application providers.

For example, a user can create a document on a table using iWork & have the cloud services to get the document processed for publication. Proof reading, Editing, reformatting, Translating etc., can be done either by machines or humans or a mix of both - all enabled by the cloud, and finally published in the cloud.

Closing Thoughts 

The growth of mobile technology has clearly changed our lives. As mobile platforms mature, and cloud infrastructure evolves, the use of cloud computing through mobiles will increase and have a much bigger and deeper impact on our lives.

Phone service providers and application providers will have to innovate on developing new tools and capabilities - which will allow users to develop new and unique apps.