Arweave and the problem of long-term data storage

Stacked Cards

The Fragility of the Internet

Sam Williams, the founder of Arweave, and the Arweave team started out with a mission to solve the long-term storage of data.

They looked at the current state of how information was kept on the internet and how people and companies managed their own files. What they saw was a very fragile system.

A few facts about the fragility of the current state of the internet:

  • Information is transient: One third of all the information on the internet is changed or gone within 2 years of it being put up; and after 20 years the majority of it has turned over.
  • Link rot is common. Link rot refers to broken links to a web page that no longer exists. If you go back to the 1998, 72 percent of the links from the internet at that time are dead. Overall, more than half of all articles in The New York Times that contain deep links have at least one rotted link. Read More
  • Study at Harvard: more than 70% of the URLs within three legal journals, and 50% of the URLs within U.S. Supreme Court opinions suffer reference rot. Read Study
  • Changing terms of service: Companies will continually change their terms of service from what you originally signed up for. Do most people know that most companies state that they own the data you upload onto their platforms?
  • High profile cases of lost data from social media giants Facebook, Instagram, Twitter, My Space and tech titans DropBox and Google.
Lost Files tweet

What this all points to is that our personal and business data is not as safe as we think it is. There are short-term implications of potential loss, but also long-term consequences of how are we actually going to pass on some of our most valuable resources to future generations.

Read a Related Article: Internet is a Collective Hallucination: the rotting of the Internet and Lose of Data

Read a Related Article: Raiders of the Lost Web: How a Pulitzer Prize finalist's 34-page essay got lost from the web

Library of Alexandria

The new Library of Alexandria

To counter the information loss that the digital world was experiencing and the increased (and historical) censorship of data, Arweave turned to the classical Library of Alexandria for inspiration.

The Library of Alexandria was founded in the early 2nd Century BC as a storehouse of the world’s cumulative knowledge. It became the largest library of its time, with as many as 400,000 scrolls including many of the world's greatest literary and scientific treasures. The Library remained in existence for more than 400 years before its decline.

Arweave seeks to resurrect the Library of Alexandria, but in this case in digital form where it will actually last!

An Archive for the Ages

To solve the problem of long-term data storage the Arweave team looked to leverage blockchain technology and innovate upon it to create permanent storage at an affordable cost.

Blockchain provided the building blocks of what Arweave needed to provide reliable long-term storage: immutability (data that doesn't change), decentralization (not controlled by third parties), and ownership (people would be in charge of their own information). There are those words again from our previous section!

However, when looking at Bitcoin there were some obvious drawbacks that would need to be overcome:

  • It is extremely energy intensive. Operating the Bitcoin network now requires the same amount of energy as a country like Austria or New Zealand every year.
  • The Bitcoin blockchain is a poor storage of data due to the expense - among other factors. As of 2021 it only holds about 340 GB, which you and I could easily store at home on a personal hard drive.
  • The number of transactions per second on Bitcoin is low - around 5 transactions per second.

Arweave needed to overcome these obstacles without sacrificing any of the decentralization or security of the blockchain. What breakthroughs were required to make permanent storage happen? How did they use blockchain to solve this?

A

Subscribe to our mailing list and get the latest ArDrive news, updates and memes.

15987