Why Object Store is ideal data storage for Media Apps?

We are seeing a tremendous explosion of media content on Internet. Today, its not just YouTube for video distribution, there are million mobile apps which dostribute media - Audio & Video content over Internet.

Users today expect on-demand audio/video, anywhere, anytime access from any device. This increases the number of transcoded copies - to accommodate devices with various screen sizes. 

Companies are now using Video and Audio as major means of distributing information in their websites. This media content is cataloged online and is always available for users.

Even the content creation is adding new challenges to data storage. The advent of new audio & video technologies is making raw content capture much larger: 3D, 4K/8K, High Dynamic Range, High Frame Rates (120 fps, 240fps), Virtual and Augmented Reality, etc.

Content creation workflow has changed from file-based workflows to cloud-based workflows for production, post-production processing such as digital effects, rendering, or transcoding, as well as distribution and archiving. This has created a need for real-time collaboration need for distributed environments and teams scattered all over the globe, across many locations and time zones

All this changes in how media is created and consumed has resulted in such massive dataset sizes, traditional storage architectures just can't keep up any longer in terms of scalability.

Traditional storage array technolofies such as RAID will no longer capable of serving the new data demands. For instance, routine RAID rebuilds would be taking way too long in case of a failure, heightening data loss risks upon additional failures during that dangerously longer time window. Furthermore, even if current storage architectures could technically keep up, they are cost-prohibitive, especially considering the impending data growth tsunami about to hit. To top it off, they just can't offer the agility, efficiency and flexibility new business models have come to expect in terms of instant and unfettered access, rock-solid availability, capacity elasticity, deployment time and so on.

Facing such daunting challenges, the good news is that a solution does exist and is here today: Object Storage.

Object Storage is a based on sophisticated storage software algorithms running on a distributed, interconnected cluster of high-performance yet standard commodity hardware nodes, delivering an architected solution suitable for the stringent performance, scalability, and cost savings requirements required for massive data footprints. The technology has been around for some time but is now coming of age.

The Media and Entertainment industry is well aware of the benefits Object Storage provides, which is why many players are moving toward object storage and away from traditional file system storage. These benefits include:

• Virtually unlimited scalability
  Scale out by adding new server node
  
• Low cost with leverage of commodity hardware
  
• Flat and global namespace, with no locking or volume semantics
  
• Powerful embedded metadata capabilities (native as well as user-defined)
  
• Simple and low-overhead RESTful API for ubiquitous, straightforward access over HTTP from any client anywhere
  
• Self-healing capabilities with sophisticated and efficient data protection through erasure coding (local or geo-dispersed)
  
• Multi-tenant management and data access capabilities (ideal for service providers)
  
• Reduced complexity (of initial deployment/staging as well as ongoing data management)
  
• No forklift upgrades, and no need for labor-intensive data migration projects
  
• Software-defined storage flexibility and management

HPE. A leading sellers of servers and Hyperconverged Systems offers several low cost, high performance solutions for Object Storage on its servers using Software Defined Storage solutions:

1. Object Store with Scality Ring
2. Lusture File System

Scality Ring Object Store is a paid SDS offering from Scality Inc which is ideal for enterprise customers.

The Lustre file system is an open-source, parallel file system that supports many requirements of leadership class HPC simulation environments. Born from from a research project at Carnegie Mellon University, the Lustre file system has grown into a file system supporting some of the Earth's most powerful supercomputers. The Lustre file system provides a POSIX compliant file system interface, can scale to thousands of clients, petabytes of storage and hundreds of gigabytes per second of I/O bandwidth. The key components of the Lustre file system are the Metadata Servers (MDS), the Metadata Targets (MDT), Object Storage Servers (OSS), Object Server Targets (OST) and the Lustre clients.

In short, Lusture is ideal for large scale storage needs of service providers and large enterprises.

What is Flocker

Flocker is an open-source Container Data Volume Manager for your Dockerized applications. By providing tools for data migrations, Flocker gives ops teams the tools they need to run containerized stateful services like databases in production. Flocker manages Docker containers and data volumes together.

VMware VIC on HPE Proliant Gen 10 DL 360 DL 560 Servers

VMware's vSphere Integrated Containers (VIC) enables IT teams to seamlessly run traditional and container workloads side-by-side on existing vSphere infrastructure.

vSphere Integrated Containers comprises three major components:

1. vSphere Integrated Containers Engine: A container runtime for vSphere that allows you to provision containers as virtual machines, offering the same security and functionality of virtual machines in VMware ESXi™ hosts or vCenter Server® instances.
2. vSphere Integrated Containers Registry:
   An enterprise-class container registry server that stores and distributes container images. vSphere Integrated Containers Registry extends the Docker Distribution open source project by adding the functionalities that an enterprise requires, such as security, identity and management.
3. vSphere Integrated Containers Management Portal:
   A container management portal that provides a UI for DevOps teams to provision and manage containers, including the ability to obtain statistics and information about container instances. Cloud administrators can manage container hosts and apply governance to their usage, including capacity quotas and approval workflows.

Cloud administrators can create projects, and assign users and resources such as registries and virtual container hosts to those projects.

These components currently support the Docker image format. vSphere Integrated Containers is entirely Open Source and free to use. Support for vSphere Integrated Containers is included in the vSphere Enterprise Plus license.

HPE Synergy Advantage for Software Defined Storage

Cloud Platforms requires massively scalable, programmable storage platform that supports cloud infrastructure, media repositories, backup and restore systems, and data lakes. HPE Syergy is an ideal plaform for such demands that provide a cost effective SDS storage solution; and introduce cost-effective scalability that modern workloads require.

HPE Synergy 12000 Frame forms an excellent platform for high performance SDS storage that scales to meet business demand. The Frame is the base infrastructure that pools resources of compute, storage, fabric, cooling, power and scalability. IT can manage, assemble and scale resources on demand by using the Synergy Frame with an embedded management solution combining the Synergy Composer and Frame Link Modules.

 The Synergy Frame is designed to meet today’s needs and future needs with continuing enhancements to compute and fabric bandwidths, including photonics-ready capabilities. Enhancements to the HPE Synergy 12000 Frame include several new capabilities:

 • Support for HPE Synergy 480 and 660 Gen10 Compute Modules.

 • Additional 2650W hot plug power supply options: HVDC, 277VAC, and -48VDC.

 • Product version to comply with the Trade Agreements Act (TAA).

 HPE Synergy is a single infrastructure of physical and virtual pools of compute, storage, and fabric resources, and a single management interface that allows IT to instantly assemble and re-assemble resources in any configuration. As the foundation for new and traditional styles of business infrastructure, HPE Synergy eliminates hardware and operational complexity so IT can deliver infrastructure to applications faster and with greater precision and flexibility.

D3940 Storage Module provides a fluid pool of storage resources for the Composable Infrastructure. Additional capacity for compute modules is easily provisioned and intelligently managed with integrated data services for availability and protection. The 40 SFF drive bays per storage module can be populated with 12 G SAS or 6 G SATA drives. 

Expand up to 4 storage modules in a single Synergy 12000 Frame for a total of 200 drives. Any drive bay can be zoned to any compute module for efficient use of capacity without fixed ratios. 

A second HPE Synergy D3940 I/O Adapter provides a redundant path to disks inside the storage module for high data availability. 

The HPE Synergy D3940 Storage Module and HPE Synergy 12Gb SAS Connection Module are performance optimized in a non-blocking 12 Gb/s SAS fabric.

Use Cases of Ceph Block & Ceph Object Storage

OpenStack Ceph was designed to run on general purpose server hardware. Ceph supports elastic provisioning, which makes building and maintaining petabyte-to-exabyte scale data clusters economically feasible.

Many mass storage systems are great at storage, but they run out of throughput or IOPS well before they run out of capacity—making them unsuitable for some cloud computing applications. Ceph scales performance and capacity independently, which enables Ceph to support deployments optimized for a particular use case.

Here, I have listed the most common use cases for Ceph Block & Object Storage.

Unified Storage with OpenStack Ceph

Open Stack Ceph Storage is a massively scalable, programmable storage platform that supports cloud infrastructure, media repositories, backup and restore systems, and data lakes. It can free you from the expensive lock of proprietary, hardware-based storage solutions; consolidate labor and storage costs into a single versatile solution; and introduce cost-effective scalability that modern workloads require.

OpenStack Cloud platforms is now becoming popular for enterprise computing. Enterprise customers are embracing OpenStack Cloud to lower their costs: License costs and eliminates the need for dedicated storage systems. From an enterprise workload perspective, there are multiple storage requirements: 

 1. High performance Block Storage

 2. High throughput File Storage

 3. Cost Effective Object Storage

 
In OpenStack, Ceph storage provides a unified storage interface for Block, File & Object Storage. This plus the ability to scale out to 1000’s of nodes makes Ceph storage a preferred SDS solution when implementing OpenStack Cloud Platform.  

 When OpenStack Solution is built on Scaleout servers like HPE Apollo platform which provides highest storage & compute density per rack, the cost economics of an on premise solution beats a public cloud offerings.

 
OpenStack Ceph Storage is designed to meet the following requirements:

 1. High availability

 2. Security

 3. Cost Effective

 4. Scalability

 5. Configurable

 6. Performance

 7. Usability

 8. Measurable

Storage Virtualization Solutions from HPE