VMware : Introducing a Federated Storage Platform

A new Federated Storage Platform

The drive of the IT industry towards cloud is making headlines almost daily, with most analysts focusing on public cloud providers. The combined market cap of Amazon, Microsoft and Google, the top three public cloud providers, is approaching six trillion dollars - almost one-third of the entire Nasdaq capitalization. This pointedly illustrates the immense potential of this opportunity for a new Federated Storage Platform.

The public cloud wins with simplicity, elasticity, a quick time to value and perhaps most importantly, a low barrier to entry, so it makes it cheap to try new projects. There is an incredible opportunity for private datacenters to deliver similar value, and re-capture hundreds of billions of market share from public clouds. Andreesen Horowitz eloquently articulated this opportunity in their thought piece "The Cost of Cloud, a Trillion Dollar Paradox." VMware's unique value is the ability to span across both private and public clouds.

In this blog, we describe what this means for VMware's strategy for data, data management and storage.

Storage challenges

Traditional storage arrays are defined by fixed hardware boundaries, which confine their resources. As a result, they are hampered by trapped data, stranded capacity, limited scalability and elasticity, complex placement decisions, limited data and compute mobility across (physical or logical) system boundaries, and high sunrise and sunset cost. They require upfront obsolescence planning with significant investments to placate technology risk, conflated by disruptive and expensive end-of-life tech forklifts with data migration sprees. In short, traditional storage array architectures cannot deliver a cloud-like experience. The industry's continued march towards the multi-cloud/zettabyte era is only exacerbating these limitations.

Hyperconverged Infrastructure (HCI) has successfully stretched system boundaries, and created a whole new industry. It provides x86 server economics, and software-defined infrastructure simplicity and flexibility. However, its easy packaging becomes limiting at scale, and it doesn't solve multi-cloud services well.

VMware HCI Mesh is a big step in the right direction to disaggregate compute and storage, but it doesn't span across clusters or create a multi-cloud surface.

Now, VMware is rethinking its software platform for multi-cloud. Storage is foundational because every business is a data business, so we'll focus on that first.

Rethinking storage

We're introducing a new concept: the Federated Storage Platform (FSP). The FSP can orchestrate data across the datacenter, across any number of vCenters and storage backends. FSP provides seamless compute/storage disaggregation, datacenter-wide data accessibility, datacenter-wide storage policy based on application needs (including global placement), datacenter-wide storage and data insights, datacenter-wide VM migration and seamless storage expansion.

As a result, the Federated Storage Platform can drive cloud-like storage operations, and deliver cloud-like self-service storage consumption based on application needs.

How does it do that?

FSP converges mixed storage systems into one single cloud-like consumption surface that can span the entire datacenter:

• Common Volumes: A new presentation layer that converges mixed physical storage resources from heterogeneous vSAN clusters, vVols storage arrays and NFS storage arrays into uniform data infrastructure, cleanly abstracted by the policy.
• Project Helios: A new cross-cluster control plane that composes the data infrastructure into elastic pools with full visibility into how it's consumed. Helios provides a cloud-like infrastructure consumption experience across heterogeneous storage types, and a common surface to mesh with other services.

A few key benefits

Datacenter-wide coherent storage policy

FSP is fully integrated with SPBM (vSphere Storage Policy Based Management), and Helios automatically manages storage pools and policies, based on application needs.

vSphere consumers can define these policies to meet their requirements, and Helios determines which of its storage types meets them best, instantiates the storage and places the data accordingly.

Holistic insights

FSP supports advanced policies like quotas or performance, for instance, to support service tiers.

FSP converges physical storage silos across the datacenter into one single global view, including all provisioned objects, policies, and so on. With its single global view, FSP has full visibility into all consumers of all of its datastores exported across all vCenters. Therefore, it can significantly extend the current management capabilities of vSphere with much better holistic resource decisions.

Seamless storage expansion

FSP can seamlessly add new storage. For instance, new storage can be added without requiring the replacement of current storage, eliminating disruptive forklift upgrades.

Helios, the cross-cluster control plane, automatically discovers and adds the new storage to the federated store. It can also load-balance to new storage by automatically orchestrating Storage vMotion. This offers advantages like eliminating hot spots, mitigating capacity pressure, and/or avoiding noisy neighbors. Helios only needs to move the storage, not the compute, since any compute can have access to any storage via Common Volumes. Helios automates the entire process on behalf of the administrators and app teams.

In contrast, a VI Admin doing this today would need to schedule Storage vMotion operations manually, which is error-prone and disruptive. FSP can lazily move and rebalance data objects in the background by driving Storage vMotions, and inform the VI Admin as equilibrium is approached or achieved. This also includes storage retirement: FSP can retire clusters that have reached their end of life by orchestrating Storage vMotion to evacuate and decommission them, all with selectable degrees of automation.

As a result, FSP can continuously roll storage beneath a stable cloud consumption surface, essentially providing perpetual storage that can dynamically and constantly evolve, based on what applications and customers want. FSP thus delivers on our vision of a cloud-like storage experience.

Integration with CSI and CNS

FSP is integrated with the vSphere CSI driver in Kubernetes, as well as CNS (Cloud Native Storage). Developers can continue to consume vSphere storage through storage classes in Kubernetes, just like they do today, and FSP now gives VI Admins full visibility into how Kubernetes applications are consuming vSphere storage across the whole of the datacenter.

vSphere with Tanzu and TKG-S workloads can currently only provision workloads on storage that is accessible to the cluster where the namespace is provisioned. FSP, in contrast, enables these workloads to access any storage in the datacenter with any policy.

Seamless workload migration

Today, compute (VMs, containers) can only be moved between vSphere infrastructures together with their entire state (including storage) through x-vMotion.

FSP, in contrast, enables compute (VMs, containers) to freely float across the boundaries of its corresponding vSphere infrastructure. It enables compute to access storage from any point in the datacenter, so compute can easily move between vSphere clusters, datacenters, or even vCenter Server instances.

Summary - scale different!

Storage for vSphere today is burdened with challenges like silos, trapped data, stranded capacity, limited scalability, cluster sprawl and forklift upgrades. They impact its ability to deliver a seamless, simple, cloud-like experience.

FSP provides a single cloud-like consumption surface for disaggregated, mixed storage for vSphere infrastructure across the entire datacenter. It is fully integrated with the vSphere stack.

This makes FSP a great step towards realizing VMware's vision of seamlessly distributing applications across multiple clouds.