Project Ensemble Preview: Platform Architecture

This is the second blog in my Project Ensemble Preview series. If you missed the first one, where I covered the API first approach that we are taking using GraphQL, be sure to check it out as well. If you are not familiar with Project Ensemble, please take the time to read the announcement blog. Very briefly, Project Ensemble will seek to unify and simplify cloud management services and data into a common data model and add intelligent and actionable business insights focused on application centric views and customized for personas who support those applications.

Before I dive in, please remember that anything discussed here is subject to change and may not make it into the final service. With that I'll share our legal disclaimer.

"This blog may contain product features or functionality that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined."

Pockets of Information

All IT organizations are confronted by a universal problem; management tools are specialized, and they do not share a common language. This is challenging even when the tools are provided by the same vendor.

For the individual practitioner within an IT domain (networking, security, development, etc.) this lack of commonality may not always be an obstacle to day-to-day operations. However, for cross-team productivity the need for quick access to and understanding of the domain-level information is critical to reduce time spent on discovering, diagnosing, and resolving application performance and availability issues.

Consider an application such as a CRM system. These can be complex behemoths with hundreds of interconnected and interdependent services. Likewise, there are multiple teams supporting this application and its components - and each of the teams likely has multiple tools for monitoring the application's performance, availability, security, capacity and more.

Thus, there exist pockets of valuable information that, in an urgent situation, need to be compiled and collated in a meaningful way. Traditionally, this results in the infamous 'bridge line' or 'situation room' where an incident manager assembles the experts for all teams that they believe need to be involved just to collect and analyze this disperse data. A lot of time has already been lost between the start of the problem and the start of problem resolution.

Now, imagine a world where teams maintain their own tools, nobody needs learn to work differently. Instead, the various data pools are connected to a single service. This service would:

1. Ingest inventories from all tools and normalize them under a federated model
2. Fetch minimal data to understand critical information from each tool about the entity
3. As needed, fetch additional information, or request appropriate actions from each tool
4. Discover and document relationships between entities as a source-of-truth
5. Provide a federated API for any tool to request information from any other tool

Having such a platform unleashes potential for so many other services and capabilities. And that is the idea behind Project Ensemble. Initially planned for inclusion in the vRealize Cloud Universal offering, it will unify all the vRealize services in a way not previously possible.

A Federated Data Model

Project Ensemble will be offered as a service and is being built upon a modern application framework of micro-services. It will not inherit legacy constructs or limitations that would constrain or prohibit integration with any cloud management tool.

At the core of the platform is a proven technology used by VMware CloudHealth Secure State, the Entity Data Service (EDS). This is a graph database which can scale large customers with over 200 million virtual machines plus related objects (around 1 billion graph entities).

As inventory is collected from each provider service (vRealize Operations, vRealize Automation, vRealize Network Insight, etc.) and constructs a non-opinionated model of the cloud inventory. Allow me to clarify, "non-opinionated" means that EDS makes no assumptions and does not specify how things should look. Essentially, EDS is not limited to VMware architecture constructs. It can (and does in CloudHealth Secure State) consume native public cloud inventory. There is no practical limitation on extensibility.

The entities in EDS are assigned a unique and canonical resource identifier which embed within them the location, type, and instance identity of the underlying managed resource. This format allows EDS and its collectors to normalize the different data models from different providers and merge the different views obtained from different providers, joining together those separate product namespaces into a common entity in EDS. The screenshot below shows an example of the identifier for a virtual machine.

While EDS provides a CMDB-type organization of the cloud infrastructure and workloads, there are other services that enhance this data. More importantly, the presentation layer (e.g. the graphical user interface) will be leveraged and extended by other products for cloud management, migration and governance.

[Link]

Notable Platform Services

I will not cover every architectural component in this blog, but the diagram above should help in understanding where the services I will explain are situated within the platform.

• Stats Service and Derived Data Service: The Stats service allows providers to integrate their own time-series metric data for each entity. The metric data is not copied or stored within EDS, but rather is fetched as needed via the providers' APIs. The Stats service also provides some forecasting functionality. The Derived Data service listens for events, alerts and notifications from the providers. It also can create platform specific metrics which are stored within the Project Ensemble data store. Essentially, these two services work together to provide metrics and observations based on trending and forecasting to feed the Business Insights service.
• Business Insights Service: This service federates the observations generated from the Derived Data service, through a template-driven framework, to create insights. Insights identify the impacts of the observations on performance, capacity, cost, security, availability and connectivity. More importantly, these insights consider the reach of the issue and the associated entities that may be impacted. I will discuss Business Insights in more detail in a future blog in this series, as this is an important capability delivered by Project Ensemble.
• Data Bus: The superhighway for data exchange, the event-driven data bus supports not only ingestion of inventory, alerts, events, metrics and more, but also provides data feeds for updates to the different models. This can be tapped into by the user through the GraphQL API to subscribe to selected change feeds in real time.
• Lifecycle Management: I will not go into depth in this blog, but this service supports on-boarding (and off-boarding) for customers to provide very fast time-to-value by deploying and configuring vRealize services to begin feeding the platform with inventory data.

Foundational Architecture for the Future

The architecture of Project Ensemble is modern, scalable, and extensible. It provides a non-opinionated, graph database to normalize inventory data along with additional services to decorate this data with actionable business insights.

Stay tuned and watch for my next blog that will preview more capabilities of Project Ensemble.

Want to Learn More About Project Ensemble?

If you are interested in being considered for the beta to provide VMware with input, please let us know here (https://www.vmware.com/learn/1243700_REG.html) and your eligibility will be evaluated.