How do we avert our impending data storage crisis?

There are those who argue that, over the last two decades, we’ve moved from an oil-centric economy to a data-focused one. While this may seem like something of an exaggeration, it’s impossible to deny that data is playing an increasing role in our day-to-day life.  And this is most apparent in the world of business and big data.

Unfortunately, with an increased need for data comes a greater need for data storage – which has created a very big problem!

The current situation

As we’ve discussed previously, data is now the driving force behind a multitude of business decisions. Additionally, an immeasurable number of valuable insights can be gleaned from the vast swathes of data that organizations are yet to analyze. In short, big data is big business. And this – coupled with increasingly more affordable technology – has seen us create data at a truly unprecedented rate.

In April 2013, 90% of all of the world’s data had been created within the previous two years. Today, the total amount of data in existence is doubling every other year. Such growth is not sustainable, but with data having proven indispensable, simply creating less of it is not an option. In order to avert this crisis, what is needed is storage media that offers storage density that is vastly superior to what is currently available. Here are the most likely possibilities along with a summary of their pros and cons:

We improve existing media

Storage media stalwart Seagate has officially manufactured more than 40,000 hard drives featuring HAMR (heat-assisted magnetic recording) technology and they plan to begin shipping them later this year.

HAMR – whereby a disk’s platter is heated prior to the writing process – has been developed in order to significantly increase hard drive’s storage capacity. This heating means that less space is needed to store data and results in hard drives that are therefore capable of achieving higher storage densities.

Potential storage capacity

Seagate has stated that HAMR tech will allow them to produce drives with more than 20TBs of storage before the end of 2019 and 40TBs by 2023. No further estimations are provided, though the company does state that they’ve already begun developing its successor, heated-dot magnetic recording (HDMR) suggesting that there is more to come.

Practicality

As we’ve stated previously, Seagate has stated that they plan to ship HAMR drives before the end of the year. These drives also use the 3.5 standard carriages. Meaning that they’ll easily insert into existing arrays. The drives also remain relatively affordable in spite of the inclusion of this new technology.

The cons

Whilst a 40TB HDD would represent a significant improvement on what’s currently available, it’s still unlikely to represent a long-term solution to the problem – unless HDMR proves capable of significantly boosting their capacity, that is.

We use our existing storage more efficiently

Sia is an example of what is, in our opinion, a unique and highly innovative potential answer to our current storage problems: it identifies unused space on various pieces of storage media throughout the globe, rents it from those users and then sells it to the general public as remote cloud storage.

Potential storage capacity

Sia’s website claims they’re an entire network of drives boats 4.2 Petabytes (4,200TBs) of storage. This may seem like a lot at first glance, but with the entire cloud currently storing just under 1,500 Exabytes (that’s 150,000 Petabytes) it doesn’t offer the kind of capacity needed to offer a real solution to our data storage crisis.

That said, 4.2 Petabytes is a considerable amount of storage that would otherwise have been wasted and Sia are also not the only company leveraging this technology. So, whilst decentralized cloud storage alone isn’t the answer we’re looking for, it’s far from ineffectual and is certainly an efficient way of utilizing existing storage space.

Practicality

As with most cloud storage, it’s easy to use and, thanks to the use of blockchain, is extremely cheap at just $2 per TB of storage.

Cons

We’ve already said that the decentralized cloud’s unlikely to offer the kind of storage capacity the world’s going to need to avert our impending storage crisis. We also know that trust in the cloud tends to diminish with each high-profile data breach so we’d expect the adoption rate to be somewhat slow.

We use something ground-breaking

It may sound like pure fiction, but DNA has already been used to store and retrieve data. In fact, DNA data storage is something that has a lot of people very, very excited.

Potential storage capacity

This is what sets DNA data storage apart from its competitors: just one gram of DNA could store 215 Petabytes of data. With storage capacities like this, it’s clear to see why many believe DNA could be the answer to our storage conundrum.

Practicality

As I’m sure you can imagine, the process of storing and retrieving data from DNA is cumbersome and, whilst it was first achieved five years ago, it’s still far from an accessible and practical means of storing data.

Whilst using DNA to create a useable piece of storage media is proving to be problematic, though, it could not only produce a device capable of storing a data center in a 3.5 inch HDD cradle but one robust enough to last for a millennium, also.

Cons

As we’ve said previously, no uniform way of reading and writing data to and from DNA currently exists. It’s also been widely reported that the task of retrieving the data itself is both a slow and cumbersome one. These, however, are not the greatest hurdles scientists face in trying to make DNA the world’s de facto storage media: that honor belongs to cost.

In 2017, data was successfully stored in and then retrieved from DNA but the cost of doing so was astonishingly high: synthesizing the data cost $7,000 and retrieving it a further $2,000. These are expected to drop significantly over the next few years but this could take as much as a decade.

Furthermore, as we’ve stated previously, reading data stored in DNA is a lengthy process and a device to house, write and read data stored in this medium is yet to be developed.