The channel’s role in helping customers manage the data deluge

One of the oldest pieces of data we have is a customer complaint about substandard copper, engraved on clay. At around 3,500 years old, it’s fascinating to see a) what the day-to-day lives of people were like millennia ago, and b) how well preserved the clay is. However, will future generations look back at the data we’re creating and feel the same way?

According to Statistica, the amount of data created, consumed, and stored worldwide is projected to increase from 149 zettabytes in 2024 to 394 zettabytes by 2028. That’s the equivalent of 394 billion 1TB storage drives worth of data – a mind-blowing number.

AI-generated content and images are adding to this mountain of data at a phenomenal rate. Enterprises are struggling to cope with the exponential growth in data storage demands. The channel is uniquely positioned to help here.

Balancing the creation and storage of data

Consider the amount of data that a single high-resolution security camera can generate while in operation 24/7. Add to this all of the data created by IT systems for security, resilience, or regulatory reasons, and it can feel an insurmountable problem for companies to deal with.

What’s more, as channel partners are aware, many organizations are only now starting to deal with AI-generated data, and it’s clear that a scalable and flexible strategy is needed to cope with the modern-day data deluge.

While it’s doubtful that future generations will look back at the data we’re saving with fascination as we do with the clay tablet, it is generally accepted that we need to keep as much data as possible in case it is needed in the future. This could be for regulatory reasons, records, or following, say, a security incident.

Dealing with the data deluge 

Here are some strategies that channel partners can help organizations implement to better handle these challenges:

• Hybrid placement and mobility: Most large organizations have a hybrid approach to cloud, with some data sets on-premises and others in the public cloud. It is key to understand the various implications in terms of cost, security, and resilience, and to deal with change as soon as an issue is identified, because as data grows, it will only become harder to move. One should always ask, “Are my decisions still valid if this dataset grows 10x?” While 10x might seem a lot, a growth rate of 40% equates to 10x growth in just seven years. It’s also important that data can be moved as easily as possible without severe egress charges.
• On-demand consumption for new requirements: on-demand consumption of on-premises storage systems can help deal with not just unplanned requirements, but also understand the current needs and growth profile of new applications and the associated data sets.
• Security and the growing need for data resilience: The rise of ransomware attacks, as well as broad regulatory requirements, has resulted in a stronger focus on data resilience. This can result in greater data growth, as more resilience generally means more copies of data and more systems to track and manage them. Look for solutions that provide cyber data resilience with a low overhead in terms of storage requirements but also with fast recovery, as resiliency requirements call for ever-lower recovery time objectives (RTO), generally beyond what traditional backup solutions can provide. 
• Sustainability: power efficiency of data storage systems, as well as their “carbon cost,” needs to be taken into account. Power efficiency should be evaluated as a function of capacity per watt and performance per watt, while carbon cost should include the entire lifecycle, including manufacturing, transport, and decommissioning.
• Long-term archiving: keep your eyes wide open when evaluating archiving and long-term data retention solutions (whether on-premises or in the cloud) as their cost profile will be very different if data is accessed (even rarely) versus completely cold. Take into account the total cost based on a realistic ratio of data being accessed. That includes retrieval costs for cloud object storage, but also off-site storage fees and physical retrieval, transport, and time to restore for media such as tape.
• The special case of video data: Video makes up an estimated 50% of the world’s data, so it makes a lot of sense to employ specific strategies to deal with it. Advances in the field of data compression, sampling, and optimization in general have helped keep the storage requirements as low as possible; data access requirements have also changed. Traditionally, a lot of video data was an archive, but advances in AI mean that people want to analyze and understand it. High-capacity flash storage can facilitate both cost-effective storage requirements while allowing data access with the performance levels required for AI on video.

How the channel can help overcome the data deluge

What enterprises need most of all to manage the data deluge is expertise and advice in order to form a clear strategy. Here are three ways the channel can add real value in relation to the challenges posed by AI and data storage:

1. Become a strategic adviser on AI and data storage, demystifying technologies and helping businesses understand the impacts that AI initiatives will have on their business.
2. Assess data workloads and AI readiness, helping customers understand data types and volumes (e.g., structured vs unstructured), AI use cases and growth projections, and recommend high-performance solutions based on needs. Consideration should be given to scalable object storage and storage-as-a-service (STaaS) solutions, which have distinct cost and scalability advantages.
3. It’s not all about AI. The trend towards exponential data growth was underway before generative AI arrived on the scene. Customers also need advice on how to maximize the efficiency of data storage. One way to achieve this is through high-density flash storage technologies that enhance storage density, allowing for higher capacity and better space utilisation. Further efficiencies can be gained through data storage technologies that utilise always-on data deduplication, which eliminates the need for post-process scanning, reducing latency and increasing performance.

The channel must remain relevant as data storage solutions evolve

Now more than ever, there’s an opportunity for the channel to play a key role in the way customers deploy data storage solutions for the AI era. The channel must stay on top of rapid advances in this space. For example, we’re now seeing innovation in the field of ceramic storage – a technology that has proven its ability to protect data - even the most mundane customer complaint - over millennia.

As such, to ensure your customers’ data storage is truly future-proof, look for data platforms that enable flexibility and are really able to tackle today’s data access, security, and resiliency requirements.