Most invention and innovation happens in small incremental steps. Someone notices a flaw in an existing product and makes a tweak to fix and improve it. People are forever trying to build a better mouse trap.
However, once in a while, there is an innovation that takes a giant step forward and completely changes the space it is in.
Arweave is one of these disruptive types of innovation. It makes a fundamental change in how data is stored, or more precisely how data is stored over the long term.
What is Arweave?
Arweave is a new type of technology that uses a form of database to store data that cannot be deleted or changed, and uses economics to incentivize people to store the data for long periods of time for the first time ever. This combination makes either public or private data permanent.
Arweave would describe itself as a “novel data storage blockchain protocol” enabling a permanent internet and creating truly permanent data storage for the first time. It is a pay-once-forever storage model.
Now if you work in the IT sector the above sentence probably makes a lot of sense. For the rest of us, we need a little help unpacking what that means.
The technology and design of Arweave and its system does not need to be intimidating to the average person. In fact, the more we understand it the better we will see how innovative and useful it really is.
So let’s dig into these definitions in detail and get a real understanding of what Arweave is.
To do this we will look at the problem Arweave is trying to solve, how the technology solves it, and who is using Arweave and why.
Let’s go!
What problem is Arweave trying to solve?
More than anything Arweave is trying to solve the problem of reliable long-term data storage. There is nothing else like this on the market.
It really is a new type of relationship to data.
We are used to thinking about storing data from short periods of time with the old USB stick to the newer cloud storage services that allow me to pay to store data on a monthly subscription basis.
However, what are the options for truly long-term data-storage? What if I wanted to store something for 20 years, or 50 years, or 200 years? What reliable options are there for that?
The perception we have is that when we upload a file to the internet or the cloud, it will be there for a long time. This really is a false perception, as the research is actually saying the exact opposite – the internet is a highly fluid entity that changes quickly over time.
Let’s look at the current state of things to see at how fragile data storage within the world wide web and cloud computing actually is.
Is the internet itself a good store of information?
When you look at the stats of long-term information capture on the internet it is quite shocking how much information gets deleted, lost or fundamentally changed. After just two years, one third of all data stored on the web has disappeared or changed so dramatically that the URL now links to something else.
Of course, all of this information does not need to be stored permanently, but what it speaks to is that the internet is by no means an archiving service, but, instead, it points to the temporary and fluid nature of the web.
In a 2014 Harvard Law Study it was found that approximately 50% of the links in Supreme Court cases did not work anymore. The authors also found that 70% of links in legal journals no longer worked as intended.
If there was a case to be made for long-term reliable archiving of data, surely the record of a country’s laws would fit into this category.
Much of the early internet has also been lost. And much of the internet of today is being lost quickly. Of course not all of it needs to be saved, but the main problem is that much of what we think is being saved actually is not.
Image from ThioJoe
Are the big companies a good store of data?
If you look at some of the bigger players in the game – Amazon, Google, Dropbox – all offer reliable short-term services, but what will they look like in 50 years from now or even next year?
Few companies that exist today will still be around 100 years from now. If they do, what will they be like? We cannot even answer what these companies will look like a year from now. What will their policies be like? How will they continue to manage the data that they have?
Endless monthly payments
In terms of data storage maybe the best question to answer is, what happens if you fail or forget to pay for the storage service? What if you failed to update your credit card information and your email has changed? When would your info get deleted? These companies are not going to keep it around for long periods of time if the user is not paying.
One tactic that people use is to assume that a platform like Facebook is saving all of their photos. Facebook is not a good archiving strategy as we only have to look at some of the social platforms that have come before.
Facebook had a predecessor named MySpace. MySpace was one of the original leading social platforms of the early internet, but lost the competition to Facebook. Recently, MySpace has admitted to losing 12 years (!) of all the music, photos, and conversations uploaded to their platform. Few tears may be shed over the loss of this deleted content, but some will be sorely missed and would have been valuable to retain.
MySpace is not alone as defunct AOL has lost tremendous amounts of articles from publishers.
And, this does not simply go for old platforms and companies. In 2019, when Google+ shut down, did they backup all of the old posts and photos on their defunct social media site before switching off the lights? Of course they didn’t. Google, like other companies, are continually changing their policies. In 2020, Google sent out a notice about limiting the amount of data and photos allowed to uploaded to Google Drive before you have to pay. It is inevitable that this decision will result in the loss of millions of photos that their users would like to keep.
The web will always have a transitory nature to it. Files we assumed would be around forever will be lost. The need for permanent data storage is clear.
Inside of Library of Alexandria – artistic rendition (Image: 21h007)
Arweave and the Library of Alexandria
With the issues of decay and loss of often important historical, cultural, and personal information on the current web the Arweave team set out a mission to create a permanent record of knowledge and history in one place.
Arweave had a vision of creating a digital Library of Alexandria.
The Library of Alexandria was a library designed in ancient times (200 BC) to create a store of universal human knowledge and understanding.
The Library was quite impressive in the first hundreds of years of existence, but much like our own internet, suffered from mismanagement and experienced a loss of knowledge over time.
Whereas the Library and (the internet) are prone to decline, Arweave solved the problem of data permanence over long-periods of time so future generations can have access to the history of the world.
How does Arweave solve the problem of long term storage?
It is easy to say that the data will be stored permanently, but how can they pull this off? What is the technology behind Arweave that makes this possible?
Now, we will try not to get too technical at this point, but we will have to go over a few terms that (trust us) will not be too painful.
The first term that we need to talk about is the word protocol. Now a protocol is simply a set of rules.
You may have even heard of it mentioned with the Internet, as in Internet Protocol.
The internet that we use today is a protocol – a set of rules that controls how it works so everybody can get what they want and the thing doesn’t fall apart.
Now, what Arweave did was come up with some new technology and their own protocol rules that would enable the information stored on their platform to become permanent.
The new technology and the new rules
The new technology: the Blockweave
The new technology blockweave is a blockchain with a twist. A blockchain is a type of database that uses ‘blocks’ to store data immutably, which means that the information does not or cannot change. When new information is added to this database all the previous information needs to be verified by a community before anything new is added to this database.
This is a very secure process for making sure the information stays the same, but it is also very time consuming and energy intensive.
Arweave reimagined this process so that the whole blockchain did not need to be verified but only a randomly previously chosen block did. The recall of the previous block makes a type of weave structure and thus Arweave calls it a blockweave as opposed to a blockchain.
The blockweave solved the problem of being able to store information in a place where it would not and could not be changed, and allowed it to be recalled quickly when needed.
But where was this information going to be stored, and how could it be assured that it would be stored for extremely long-periods of time?
With this twist on the technology that made the blockweave, Arweave was now able to write a different set of rules and incentives for the people who store the data and to ensure they did for long-periods of time.
Let’s look more closely at the people who are storing this data.
Who is going to store this information for long periods of time?
As we talked about the above problem of who is actually going to store this information over long-periods of time it cannot be trusted to any one single entity like an Amazon or Google. We cannot be sure they will remain popular or if they will even exist 50 years from now.
Who could store all of this information over the years? Well, the solution is that Arweave needed to have a network of 1000s and 1000s of independent people who will store this information over time.
This is the process of decentralization, where the information is distributed amongst many to be responsible for it, as opposed to centralization where you are simply trusting one entity to care for it. Decentralization is much, much more reliable for long term.
In the world of protocols and blockchain technologies, the people who store the data are called miners. These miners host the data on their private servers, nodes or create pools of miners to store the information that gets put onto the blockweave.
Why would a miner store information from Arweave on their server?
Miners store information on their servers because they are incentivized to do so through payment. Just like you wouldn’t go to work unless you got paid, they will not store it on their servers unless they get paid.
They have costs to maintain to run their servers, and if they are paid more than these costs they will host the data that is given to them.
So how does Arweave manage to pay these miners to store data over such long time periods?
Arweave uses a fee structure where users pay once to store information forever instead of being bound to monthly or yearly subscription fees. This is also a true innovation.
The natural question that comes up is, with a one-time payment wouldn’t the payment need to be extremely large to cover the cost over such a long-period of time?
It would seem this was the case from what we know about how things get more expensive over time. However, what is different about storage costs is that unlike most of the rest of life, storage prices are going down year-after-year, and not by a little, but by a lot.
The average storage prices for data have decreased by 30.5% over the past 50 years.
Arweave designed a system so that all of the costs paid to host data onto the Arweave network go into an endowment that will be able to pay for the cost of storage based on the interest generated from the endowment.
Think of an endowment for a University or non-profit. The endowment will use the interest from their pool of savings to pay the operating costs of their organization. Unlike most Universities, which have ever-rising costs, Arweave is paying for storage costs that are declining rapidly. Even in the unlikely case that data storage costs remain as they are, the one-time storage fee cover the cost of data storage for 200 years.
So how much is the one-time payment that is required?
The one-time payment is still very, very affordable.
Generally, you will end up paying around one cent for 4-6 MB of permaweb data.
For instance, this would workout to be about one tenth to one hundredth of a cent for smaller documents (50 to 250 kb), or about 2 cents for a larger audio file (10 MB).
When compared to other forms of data storage Arweave will look more expensive on an initial basis, but if you look at the price over time then even after just a few years Arweave will always play out as the low-cost option for data storage.
What is the permaweb?
The permaweb is the combination of blockweave technology and the innovative incentive system for miners. Arweave describes the Permaweb as:
“… a global, community-owned web that anyone can contribute to or get paid to maintain.
The permaweb looks just like the normal web, but all of its content – from images to full web apps – is permanent, retrieved quickly, and decentralized – forever. Just as the first web connected people over vast distances, the permaweb connects people over extremely long periods of time.”
This is truly a step forward not only for data storage, but how the average person will interact with data. It is a powerful tool that has only started to be used, and many innovations will be made with it.
Permaweb URLs look like a long string of random letters and numbers with arweave.net in the center.
This unique web address works as a permanent link to the uploaded file. So, how do we get something onto the permaweb?
How do you pay to get something onto Arweave and the permaweb?
Everything that gets put onto Arweave needs to be paid with the AR token.
Rather than being tied to a single national currency, Arweave created its own digital currency called an AR token. Any data that is put onto the permaweb needs to be paid with the AR token to be put on there. This is also what the miners receive for hosting the data on their servers.
There are many applications being developed on top of Arweave that will allow people with all different types of data needs to be able to use this technology.
But before looking at these applications, let’s look at a few cases which actually need long-term data storage.
Who needs data permanence?
Well, the uses for data permanence will probably explode as it becomes more widely known. However, we can already see a lot of different types of data being stored on the blockweave.
Individuals are already adding the following to the permaweb:
- Family history – music, photos, documents
- Digital art
- Old movies
- Books in the public domain
- Important documents
- Podcasts
- Blogs
From a business, organizational or cultural side the number of use cases seems to be infinite.
A few that stand out are:
- Permanent storage for other blockchain technologies
- Legal documents (like Supreme court records)
- Government data such as Land Titles or Records
- Architectural Drawings
- Academic Papers and Research
- Newspaper and blog articles
- Financial Ledgers
The one last thing we want to explore about Arweave is not necessarily what can be uploaded to it but what can be built on top of Arweave. This is one of the most exciting developments of all.
What can be built on top of Arweave?
Already there are over 300 applications built upon the system.
At first they were simple apps like a chess application or a to do list. However, over the past year a number of exciting business communities have launched applications that will make real differences in the everyday lives of people and businesses.
For instance,
- ArDrive – file sharing app like Dropbox
- Spheron – the ability to put whole websites onto Arweave
- Pianity – music sharing and NFT application
- Redstone – customizable defi data sets
- Glass Protocol – video and livestreaming
These applications and others bridge the gap between Arweave and the average person.
They allow the common person who may never know what Arweave is the ability to keep their information in a secure and permanent way.
In the future, the permaweb is likely to extend to all areas of life. There are going to be apps for everything from family trees to helping banks keep permanent records to facilitating business to business applications.
Disruptive inventions tend to set off a chain reaction of further innovations as the product ripples out into the culture. It will be exciting to see what actually gets built on top of Arweave in the next 5,10, 50 years and beyond!
August 2021 Update: Learn more through this new video from Coin Bureau