Understanding Data, Analytics and Technology platforms Part 1

Note: This article was co-authored with Saraswathi Ramachandra

Today, lot of young engineers often ask me about analytics.

Often, the word "Analytics" is used in conjunction with "Big Data" or "Hadoop". But in real business world, Hadoop or bigdata forms a small portion (but growing) of today's business analytics.

This prompted us to write this blog to educate folks on realities of analytics in business today. This blog is written in three parts for ease of reading.

Part-1: Understanding Data for Analytics
Part-2: Understanding different Uses of Analytics
Part-3: Understanding Analytics Technology Platforms

Part-1: Understanding Data for Analytics

Before we start talk about analytics, we need to understand data. What kind of data is used in business analytics?

Today business have a wide range of data sources that are used in business analytics. The diagram below shows the popularity of data types.

Today in business organizations, Structured data still rules. According to a research done by TDWI Research in 20151, Structured data is used by 90% of all companies they surveyed across 4 continents, covering 357 large enterprises across all categories.

Figure-1: Different Data Types & Popularity

Organizations are trying to bring together multiple disparate data sources. For now, many of those data sources are the traditional data types such as structured data in Databases, data warehouse, legacy data, data from previous reports. These data sources are typically well curated and are well understood by businesses.

However, new data sources such as unstructured data (which is mainly audio, video streams), data from IoT device, Point-of-Sale machines, Social media,  Telemetry data, Web logs, click streams, etc are becoming more relevant and important for businesses - as they help businesses get a real-time view of things happening to their business. Therefore the use & value of these data types are increasing rapidly.

However, new data sources such as unstructured data (which is mainly audio, video streams), data from IoT device, Point-of-Sale machines, Social media,  Telemetry data, Web logs, click streams, etc are becoming more relevant and important for businesses - as they help businesses get a real-time view of things happening to their business. Therefore the use & value of these data types are increasing rapidly.  For instance, close to 31% of respondents cited geospatial data as being in use today. Geospatial data can be very useful for real-time analytics. Airlines use geospatial data to track and simulate the flight paths of thousands of flights a day. They can view all of these flights on an interactive map. If weather or other events occur, air traffic dispatchers can make tweaks to flight paths to as part of air traffic control. This can help reduce costs.

Even use of click-stream data, as well as machine-generated data from sensors, data from social media are being used by business to understand customer in real time. Data is continually processed and if it exceeds a certain threshold based on a calculation, an alert is generated or a downstream application is notified. This allows businesses to respond in near real time to customer activities. Today, companies are using very sophisticated technologies to analyze data in real-time and manage customer experiences.

Recently, streaming data from sensors and other IoT devices are being analyzed at the source of data generation to create time-series data. (This is often referred to as analytics on the edge), time series data can then be stored in databases & used for further analysis.

Coming Next: Part-2: Understanding different uses of analytics

10 Best Practices for building Analytics based Organization

Business analytics is the key for decision making in organization at all levels. Startups and disrupters are using data analytics - both traditional BI & Big data tools for decision making, and in many cases, automating decisions. Yet, there are several companies that are yet to fully embrace data analytics.
In our opinion, here are top 10 practices that must be followed to transform any organization into a data driven organization.

1. Start with Top Executives
Top executives must be the first ones to adapt analytics. Once top executives embrace data analytics, the rest of the organization will follow. Organizations can jump start this process by hiring a Chief Analytics Officer, who can create a data driven executive teams at the top, and then guide rest of the organization in adapting analytics. This will enable an organic growth and will take time.

2. Start Analytics Project which impacts Revenue
The first of analytics projects that are taken up must have a strong connection to bottom line. The output of the analytics must be connected to revenue or profits or costs. Projects which have such a deep & big impacts will have a higher likelihood for making a business impact and for being successful. Early success is essential to take analytics into other parts of the organization.

3. Consider organizational implications
Often in older, legacy bound organizations, cultural issues are the hardest to overcome. Even with executive sponsorship and proof of concepts to illustrate the value of analytics, there will be reluctance to adapting new technologies. Start with communicating key accomplishments & push the analytics message and continue to evangelize. This will take time, but with patience it can be done.

4. Think about Organizational design
Organizational structures in legacy companies are not designed for analytics based workflow. Hence it becomes important to reorganize to take full benefit of analytics led workflows and business processes. Reorganization will also accelerate adoption of analytics in business operations.

5. Consider new talent from outside the organization
Though it makes sense to build all the required talent in-house via training, Legacy organizations may need talent from outside the organization to build analytics skills. Alternatively, one can also consider 3rd party consultants to help execute analytics projects and accelerate deployment of analytics.

6. Move rapidly towards deployment
Analytics are valuable only when it is deployed to make decisions. Proof of concepts are good, but will be useless if it is not deployed. This means teams must organize to execute around these analytics. Successful organizations build rapid deployment of analytics.

7. Take control of your Data
Data analytics needs strong data governance & stewardship. Data management, protection and data quality - which goes a long way in building a trusted data pool (or data lake) which then can be used for analysis.

8. Track Outcomes
Track outcomes of all data analytics projects. During the initial stages, there will be missteps, constant debate on what the right thing to do. Having complete transparency about analytics projects and its outcomes helps accelerate adoption

9. Invest in new tools & Technology
Data analytics tools are rapidly evolving. New tools are being released all the time - which brings in new capabilities to create analytic models, faster analytics and AI based automated decision making.

10. Rapidly Adapt advanced analytics
Advanced analytics such as deep neural networks, robotics & automation are rapidly maturing. The advanced analytics tools work on unstructured data, streaming data, and telemetric data. These new types of data analytics will transform organizations. 

Fintech - Success Factors for Innovation

As a technology disrupter, challenging an entrenched incumbent, Fintech products must be innovative solutions. But just being innovative is not enough - for example, developing a new way to make cashless payments is not enough. Success depends on several other factors.

There are three common factors behind every successful technology product. Working judiciously on these factors can create a successful product. Now, lets take a look at those three factors.

Factor 1. Think 10X

To be successful against an entrenched incumbent, one has to be truly innovative. One way to be innovative is to think 10X. For example, if it takes 120 seconds to make a payment with a credit/debit card, a cashless payment system must be able to do it in under 12 seconds! In the second iteration of the product, the transaction times must further decrease by 20% , i.e., under 10 seconds and so on.

10X factor must be applied to all aspects of business. For example, how can the business grow 10X in 1/10th the time taken by the incumbent?

Unless one does not aim high and deliver on it, customers will not switch to the newer technology.

10X formula in business forces one to rethink every aspect of business, and forces innovators to solve more complex challenges of tomorrow today, instead of incrementally solving today's problems.

Factor - 2. Be very agile after product launch 

"No Battle Plan Survives Contact With the Enemy"  said the German military strategist Helmuth von Moltke. The same is true when launching a new & innovative products. A product is always designed with certain assumptions and all those assumptions must be verified on the ground - when customers are using the product.

Product development teams must be agile to respond to the customer feedback, learn what works (or doesn't) and make changes rapidly in response. A smaller "beta" release helps to make rapid iterations of the product based on customer feedback and never stray too far from what market really wants.

There are no holy cows or egos to massage. If a feature is not liked by customer, be ready to drop it. Remember that the entrenched competition cannot move as fast as the challenger, and Agility is the key for success.

Factor 3. Share all knowledge

Collaboration is essential for innovation. Collaboration happens best when everyone shares information openly and discuss it freely. In case of Fintech, there are several points of data collection and this data & the analysis must be shared openly within the organization - so that the entire organization can act on that information & be agile.

Remember, innovation thrives in a culture of openness.   

Fintech - The Success Factors

In today's changing world, for Banks to remain competitive, Banks must commit to transforming themselves into fully-digitalised businesses and must operate like a true technology driven company.  

From our perspective, 10 key aspects that have defined success in Fintech companies that Banks needs to adopt -

1. Customer experience - Fintech companies constantly focus on improving customer experience and learn from every interaction. This is the mantra of success.

2. Digitalization - Create a "stretch vision" for digitalization blue print and make CxO's accountable for it. The "stretch vision" are based on what digital outcomes and it should not be constrained by digital interactions or by current technologies or by as-is scenario. These stretch visions should become inspiration for new innovations.

3. Technology is a key enabler – Fintech's are changing the world through agile platforms and new technologies. Banks must acquire new capabilities and technologies and need to invest in promising technologies. Banks must be willing to take on technology risks and bet on emerging technologies to build new parallel solutions/services - instead of the relying only on traditional ways of relying on "Tested & Proven" solutions.

4. Talent Management - Banks must invest on right talent, if needed on a global scale. Companies have to hire people from other regions/nations & en masse. For example, Indian banks should not shy away from hiring people in Silicon valley. Banks should note that they are hiring digital skills not industry experience. To succeed, hire the right talent and change internal staffing processes to do so – if required.

5. High Performance Culture - Success in new technology development depends on having the right digital talent. Key employees must be protected from "business as usual" attitudes and teams need to be built on high performance attitude. This implies changing existing HR polices,  benefits & operating models etc. For some banks, it makes sense to up a separate business unit that nurtures Fintech initiatives. This unit operates outside the "Traditional Banks" and does not suffer from traditional organizational hierarchies and limitations. in order to increase collaboration, productivity and "mind-shift".

6. Data driven decisions - Banks must shift their mode of operations from process driven to data-driven. Data analytics drives every activity and provides insights to decisions. Data analytics and modelling will have to continually evolve and keep adding to its value proposition. Banks must embrace live testing in Fintech as they adopt agile development and work in "live-beta" environments - in order to increase the pace of innovation and acceptability.

7. Create Eco-System through collaboration - Banks must build collaboration, partnership, and open transactions to succeed. This essentially calls for a big mind-shift as Banks now need to look at the entire ecosystem and collaborate with the best in the world to create a global value chain. New regulations impose these in forms but this a great way to successes and this helps compete/collaborate with startups.

8. Agility with Zero-waste attitude- Banks must learn to manage development projects like a start-ups with complete agile process. Along with agility, zero-waste based budget approach for technology development is key. All expenses must be aligned with the value creation and not based on effort spent.

9. Scalability – Solutions built must be scalable across the organization and value chain. This implies that all new technologies being developed must have scalability built into its core DNA and solutions developed may be able to rapidly scale for success. Startup fintechs today keep scalability as key requirement and don't face downtime issues. Banks need to fast track this process. For example, PayTM was able to scale up by more than 1000% in one month after demonetization in India seamlessly.

10. Challenge Status Quo - Banks must challenge everything. Status quo is not acceptable when dealing with Fintech which impacts all functions, products, business units and locations. Banks must examine & change (as needed) all aspects of the business: embracing both customer-facing and back-office systems and processes and be more innovative.

What is driving Fintech?

Fintech has taken the banking world by storm. Customers are embracing Fintech services rapidly and wholeheartedly. Digitization and Fintech is impacting all aspects of banking, investments & finance. Fintech's impact will be huge and will fundamentally transform the finance industry.

Today, I had open discussion with a bunch of managers at HPE, a group of really smart folks with advanced degrees from tier-1 colleges, on what is driving the rapid move towards Fintech and here is what came out of this discussion.

Digitalisation is a Mega trend 

Digitalisation is happening everywhere and is going far deeper than the cost-saving potential from innovative IT or even developing new revenue streams. Digitisation is fundamentally changing people behaviour.

Digitisation is being driven by three main forces

1. Customer experience
2. Technology enablement
3. Cost Savings

1. Customer experience

Customer experience is really driving Fintech adoption. Customers love the ease of using mobile, web for banking interactions - when compared to walking into a physical bank branches. The ease and the enhanced customer experience of using technology solutions over the good old way of interacting with banks/insurance/finance companies.

Customers are really pushing banks towards  digitalisation. Customers are the leaders in embracing Fintech and are dragging banks into it. Today, customers expect a seamless multi-channel experience - be it from a physical branch or on a mobile app, A consistent service levels that are of global standards.

Customers who got used to the convenience of Fintech are judging banks in how well banks are able to meet their needs. For example, it takes 8 distinct steps to transfer money online when compared to 2 steps in PayTM.

2. Technology enablement

Smart phones and Mobile Internet is rapidly influencing customer behaviour. The ease of connectivity with 4G technology and increased bandwidth has enabled customers to access technology enabled services.

Technology is no longer the realm of large banks and upper class. Customers of all social classes are now very comfortable using mobile, Internet technology and are using it everyday. During the demonitization exercise in India in November 2016, people from all walks of life started to embrace Fintech - right from street vendors, small shop owners, handy men like plumbers etc.

Technology has empowered people and people learnt to harness the power to technology to their benefit.

3. Cost Savings

Unlike technology developments of the past where banks embraced new technology to lower costs, today customers are embracing new technology to save costs for themselves. Customers are unwilling to pay the bank transaction charges for sending money or for loan approvals etc. Customers are using Fintech to reduce their cost of using financial services.  For customers there are two forms of cost savings - time & fees. Fintech provides both benefits, it lowers transactional costs to near zero, and gives instantaneous service which saves time.

Today, Banks are at a disadvantage with their high cost structures and customers are tired of paying banks for basic financial services. Hence it is cost savings to customers that is driving Fintech.

Closing Thoughts 

Fintech can cut costs by up to 90% while rapidly improving turnaround times.  Digitalisation customers to do more with their money and increases the velocity of transactions. This implies that banks need to evolve and rapidly embrace Fintech to make best use of it - allow customers to interact often, provide new innovative services and create better customer experiences. Else customers will move to those banks or new age finance companies that gives them what customers need.


To succeed in the world of digitalisation, Banks must develop a clear strategy that puts customers first - ahead of every single internal processes and costs. This may require redsigning current processes - while looking holistically at end-to-end customer experience, and must go beyond basic regulatory requirements.

The goal must nothing short of "total customer delight", and must traverse across channels for the customer - as a continuous engagement with customers across partners in financial services.