Data Storage Tiers and HPE Solutions

Data centered economy implies that organizations need to store vast volumes of data, and this data has to be stored in an economic way - so as to get best performance/cost ratios. This implies creating multi-tier data storage systems, and having automated data management systems - such as HPE DMF solutions.

Automating Data management system over multiple tiers and building a cost effective SDS  systems is the way forward.

Today, HPE offers several SDS solutions for each data storage tiers.

Tier-1: VMWARE vSAN, HPE Simplivity & HPE VSA 
Tier-2: Lusture SDS with ZFS for scaleout NAS storage 
Tier-3: Scality Ring Object Store
Tier-4: HDD based RDX Cartridges for long term off-site secure backup

Tier 1 : This is frequently accessed storage, best build on SSD drives. NVMe SSD drives offer highest IOPS and throughput, SAS SSD offers highest capacity - while offering very high levels of IOPS performance.

Tier-2 :  Frequently used data, often files used by individuals. These files are best stored in a scale out NAS system. HPE offers Lusture ZFS built on Apollo 4500 servers - which offers infinite storage capability and at low cost solution when compared to dedicated NAS arrays.

Tier-3 :  Archived Object Store data, used infrequently but has to be stored for business purposes. HPE offers Scality Ring Object Store solution built on Apollo 4500 servers - which offers infinite storage capability and at very low cost solution  - which does not need backup, as data is replicated across multiple datacenters. 

Tier-4 : Backup Data stored in secure off line, off site location. Historically, tape storage was used for this backup, but with low cost HDD and very high reliability of Hard Drives, companies can use HDD for backup storage. HPE offers FLX HD cartridge solutions for long term data backup and off site archival.

HPE Elastic Platform for Big Data Analytics

Big data analytics platform has to be elastic - i.e., scale out with additional servers as needed.

In my previous post, I had given the software architecture for Big Data analytics. This article is all about the hardware infrastructure needed to deploy it.

HPE Apollo 4510 offers scalable dense storage system for your Big Data, object storage or data analytics? The HPE Apollo 4510 Gen10 System offers revolutionary storage density in a 4U form factor. Fitting in HPE standard 1075 mm rack, with one of the highest storage capacities in any 4U server with standard server depth. When you are running Big Data solutions, such as object storage, data analytics, content delivery, or other data-intensive workloads, the HPE Apollo 4510 Gen10 System allows you to save valuable data center space. Its unique, density-optimized 4U form factor holds up to 60 large form factor (LFF) and additional 2 small form factor (SFF) or M.2 drives. For configurability, the drives can be NVMe, SAS, or SATA disk drives or solid state drives.

HPE ProLiant DL560 Gen10 Server is a high-density, 4P server with high-performance, scalability, and reliability, in a 2U chassis. Supporting the Intel® Xeon® Scalable processors with up to a 68% performance gain1, the HPE ProLiant DL560 Gen10 Server offers greater processing power, up to 3 TB of faster memory, I/O of up to eight PCIe 3.0 slots, plus the intelligence and simplicity of automated management with HPE OneView and HPE iLO 5. The HPE ProLiant DL560 Gen10 Server is the ideal server for Bigdata Analytics workloads: YARN Apps, Spark SQL, Stream, Mlib, Graph, NoSQL, kafka, sqoop, flume etc., database, business processing, and data-intensive applications where data center space and the right performance are of paramount importance.

The main benefits of this platform are:

1. Flexibility to scaleScale compute and storage independently
2. Cluster consolidationMultiple big data environments can directly access a shared pool of data
3. Maximum elasticityRapidly provision compute without affecting storage
4. Breakthrough economicsSignificantly better density, cost and power through workload optimized components

Why Use Lusture File System?

Lusture File System is designed for a large-scale, high-performance data storage system. Lusture was designed for High Performance Computing requirements – which scales linearly to meet the most stringent and highly demanding requirements of Media applications. 

Lustre file systems have high performance capabilities and open source licensing, it is often used in supercomputers. Since June 2005, it has consistently been used by at least half of the top ten, & more than 60 of the top 100 fastest supercomputers in the world, including the world's No. 2 and No. 3 ranked TOP500  supercomputers in 2014,Titan & Sequoia. 

 Lustre file systems are scalable and can be part of multiple computer clusters with tens of thousands of client nodes, hundreds of petabytes (PB) of storage on thousands of servers, and more than a terabyte per second (TB/s) of aggregate I/O throughput.  This makes Lustre file systems a popular choice for businesses with large data centers, including those in industries such as Media Service, Finance, Research, Life sciences, and Oil & Gas.

Why Object Store is ideal data storage for Media Apps?

We are seeing a tremendous explosion of media content on Internet. Today, its not just YouTube for video distribution, there are million mobile apps which dostribute media - Audio & Video content over Internet.

Users today expect on-demand audio/video, anywhere, anytime access from any device. This increases the number of transcoded copies - to accommodate devices with various screen sizes. 

Companies are now using Video and Audio as major means of distributing information in their websites. This media content is cataloged online and is always available for users.

Even the content creation is adding new challenges to data storage. The advent of new audio & video technologies is making raw content capture much larger: 3D, 4K/8K, High Dynamic Range, High Frame Rates (120 fps, 240fps), Virtual and Augmented Reality, etc.

Content creation workflow has changed from file-based workflows to cloud-based workflows for production, post-production processing such as digital effects, rendering, or transcoding, as well as distribution and archiving. This has created a need for real-time collaboration need for distributed environments and teams scattered all over the globe, across many locations and time zones

All this changes in how media is created and consumed has resulted in such massive dataset sizes, traditional storage architectures just can't keep up any longer in terms of scalability.

Traditional storage array technolofies such as RAID will no longer capable of serving the new data demands. For instance, routine RAID rebuilds would be taking way too long in case of a failure, heightening data loss risks upon additional failures during that dangerously longer time window. Furthermore, even if current storage architectures could technically keep up, they are cost-prohibitive, especially considering the impending data growth tsunami about to hit. To top it off, they just can't offer the agility, efficiency and flexibility new business models have come to expect in terms of instant and unfettered access, rock-solid availability, capacity elasticity, deployment time and so on.

Facing such daunting challenges, the good news is that a solution does exist and is here today: Object Storage.

Object Storage is a based on sophisticated storage software algorithms running on a distributed, interconnected cluster of high-performance yet standard commodity hardware nodes, delivering an architected solution suitable for the stringent performance, scalability, and cost savings requirements required for massive data footprints. The technology has been around for some time but is now coming of age.

The Media and Entertainment industry is well aware of the benefits Object Storage provides, which is why many players are moving toward object storage and away from traditional file system storage. These benefits include:

• Virtually unlimited scalability
  Scale out by adding new server node
  
• Low cost with leverage of commodity hardware
  
• Flat and global namespace, with no locking or volume semantics
  
• Powerful embedded metadata capabilities (native as well as user-defined)
  
• Simple and low-overhead RESTful API for ubiquitous, straightforward access over HTTP from any client anywhere
  
• Self-healing capabilities with sophisticated and efficient data protection through erasure coding (local or geo-dispersed)
  
• Multi-tenant management and data access capabilities (ideal for service providers)
  
• Reduced complexity (of initial deployment/staging as well as ongoing data management)
  
• No forklift upgrades, and no need for labor-intensive data migration projects
  
• Software-defined storage flexibility and management

HPE. A leading sellers of servers and Hyperconverged Systems offers several low cost, high performance solutions for Object Storage on its servers using Software Defined Storage solutions:

1. Object Store with Scality Ring
2. Lusture File System

Scality Ring Object Store is a paid SDS offering from Scality Inc which is ideal for enterprise customers.

The Lustre file system is an open-source, parallel file system that supports many requirements of leadership class HPC simulation environments. Born from from a research project at Carnegie Mellon University, the Lustre file system has grown into a file system supporting some of the Earth's most powerful supercomputers. The Lustre file system provides a POSIX compliant file system interface, can scale to thousands of clients, petabytes of storage and hundreds of gigabytes per second of I/O bandwidth. The key components of the Lustre file system are the Metadata Servers (MDS), the Metadata Targets (MDT), Object Storage Servers (OSS), Object Server Targets (OST) and the Lustre clients.

In short, Lusture is ideal for large scale storage needs of service providers and large enterprises.