Over the years, the grand contours of data and data management have evolved constantly to provide new ways of addressing an ever-changing set of requirements.
From relational databases to data warehouses, data lakes and now lakehouses, structured or unstructured data, in various formats, on-premises or in the public cloud, and accessible via many different protocols, the result for many organisations is that they have dozens of ways to manage and address data to support all their applications, business units and populations of users.
The introduction of strict data regulations as well as a greater focus on data resiliency, security and governance mean that analysing and operationalising this data while complying with regulatory requirements and internal policies has become a very complex task.
This is compounded by the rise of AI in the Enterprise, which not only requires access to data in all its forms in order to deliver its promises, but also generates its own data that needs to be understood, stored and secured.
Siloed storage is inefficient, vulnerable and costly
How does data storage fit in? In many large organisations, on-premises storage arrays are still dedicated to vertical application stacks, not just isolated but also managed independently on an array-by-array basis.
There are generally very good reasons for this mode of operations, which could be technical or organisational. The result however, is fragmentation and silos of data and management which is not only inefficient and costly (as it can often lead to underutilisation) but can also hinder the access to or the movement of data.
This creates a situation in which data footprint grows, and at the same time becomes increasingly inaccessible, harder to manage, and more vulnerable to compliance risk and cyber-threats. That’s because, as data is attached to particular applications, it becomes captive with no way to access it other than via its native application, or to copy it and move it.
Developers, data scientists and users need to work on datasets as they build and re-work applications, run reports and analytics etc. in real time. Without being able to work on them in one place, they copy data to a convenient location. That’s a common practice that takes data beyond the reach of other applications and beyond best practice governance. The result is multiple exposed copies over which the organisation loses visibility and control, lacking the necessary governance and protection.
If an application, other than the one for which the storage array and its volumes was provisioned, wants to access the data, this kickstarts a considerable process which brings with it additional storage management overhead and the possibility of the dreaded data migration.
In summary, the customary attachment of storage arrays to particular application stacks results in captive silos of data on one hand, and the uncontrolled and ungovernable proliferation of data on the other.
The impacts are felt in increased costs in terms of management overhead and storage capacity, increased vulnerability to cyber-threats due to a greater surface of attack, and a corporate body of data that is more difficult to access with analytics and AI to gain insights.
Have Hyperscalers shown the way?
The obvious contrast to all this is how things work in the cloud, and especially among the hyperscalers. In those environments, storage is abstracted, with capacity pooled by performance tier, and managed by a unified control plane layer. It certainly makes this capacity easy to consume for the hyperscaler’s customers, but it also results in very efficient resource utilisation for the hyperscaler itself.
Regardless, the challenge is to bring the abstracted and pooled model of operations to the enterprise and to transition from storage management to data management so that, instead of managing individual storage systems, storage is managed as a unified fleet, consistently and intelligently across the whole environment which enables the organisation to understand and pilot the data sets independently of the hardware that underpins it.
Abstracted, policy-based, API-driven
The next evolution in the enterprise datacentre will be towards such a virtualised cloud of data. That can be enabled by the right storage technology and intelligent software that can harness all of the resources under its control and enable the so-called “cloud operating model”.
A virtualised data cloud needs a unified control plane, with intelligent, autonomous data management and governance across the entire information system, whether on-premises, in the public cloud, or across both.
API driven and policy-based, it enables self-service, consistency of operations, integration with the entire application landscape and easy governance regardless of the scale of the information system while eliminating silos. That stands in stark contrast to the still very commonplace scenario of array-by-array management and manual operations.
On the hardware side, capacity and performance must be able to scale dynamically and transparently, without downtime, performance degradation or unnecessary data migrations. It must also allow the organisation to offer various classes of services matching its policies in terms of performance, resilience, security and compliance that can easily be consumed by applications, business units and internal customers, all orchestrated by the distributed control plane.
Data can then easily be shared, copied and distributed in a policy-driven, automated and efficient manner to match the requirements of modern applications and AI use cases.
How to answer the challenge
So far there’s only one architectural approach that has been able to tackle this challenge; what we at Pure Storage define as the Enterprise Data Cloud (EDC).
EDC brings together a number of core pillars of functionality to create a unified data plane with the aim of providing a consistent experience across block, file and object in the customer storage fleet, alongside advanced data services such as snapshots, replication, multi-site high-availability and anti-ransomware features.
These capabilities are enabled by a common operating environment across array models. On top of this, intelligent insights and fleet-wide optimisation result from integrated AIops monitoring and analytics.
Next in the stack comes a unified control plane across pools of storage with different classes of data managed by policy. This allows data to be provisioned, according to workload profile, and move as required between compatible classes of storage. The result is data management that puts the emphasis on policies, automation and APIs to do the heavy lifting, not manual operations and array-by-array management.
Finally, a Storage as-a-Service consumption model and non-disruptive upgrades allow customers to scale and evolve their storage fleet while ensuring infrastructure remains current and agile.
The way forward
The Enterprise Data Cloud solves the challenges of fragmented, siloed storage by unifying data management across block, file, and object storage. With policy-driven automation, intelligent insights, and seamless scalability, EDC transforms storage infrastructure into a software-driven, AI-ready platform that ensures agility, governance, and operational efficiency.
In short, the intention is that customers can stop managing storage and start managing data.
Discussion about this post