Data Revolution: A Comprehensive Reveal of Decentralized Storage
TechFlow Selected TechFlow Selected
Data Revolution: A Comprehensive Reveal of Decentralized Storage
As of June 2023, the total storage capacity of decentralized storage has exceeded 22,000 PB, while network utilization is only about 20%.
Navigating Web3 tides with focused insights
TL;DR
Decentralized storage allows individuals or small groups to contribute their idle storage space as units within a storage network, bypassing the absolute control that centralized entities like AWS and Google Cloud have over data.
Low storage costs, data redundancy backups, and token-based economics are also hallmarks of decentralized storage, with numerous Web3 applications built upon this foundational infrastructure.
As of June 2023, the total storage capacity across decentralized networks has exceeded 22,000 PB (petabytes), yet network utilization stands at only around 20%. This indicates significant room for future growth.
Among existing capacities, over 80% is provided by Filecoin, making it the undisputed leader in the field. Filecoin has also launched initiatives such as Filecoin Plus and FVM to incentivize developers and promote ecosystem development.
With the rise of artificial intelligence and fully on-chain gaming, the decentralized computing and storage sectors are poised for exciting growth opportunities.
1. Why Do We Need Decentralized Storage?
Cloud storage services like Dropbox and Google Cloud have revolutionized how we store and share large files (such as videos and photos) online. They allow anyone to store terabytes of data at far lower costs than buying new hard drives and access those files from any device when needed. However, there's a catch: users must rely on centralized entities' management systems, which may revoke account access at any time, share files with government agencies, or even delete files without cause. This model leads to unclear ownership of data assets and effectively grants monopolistic power over data to major internet companies like Amazon and Google. Moreover, downtime from centralized services often brings catastrophic consequences.
The storage domain is inherently suitable for decentralized applications. First, it addresses user concerns about data privacy and ownership. Files stored on decentralized file services are not subject to influence from any central authority—such as governments that might want to control or censor content. It can also prevent private companies from actions like service censorship or sharing files with law enforcement.
Second, storing massive amounts of data inherently requires distributed systems for indexing. Even existing centralized cloud services use distributed solutions such as Spanner and TiDB. While distributed does not necessarily mean decentralized, decentralization always implies distribution. Unlike centralized architectures, current decentralized approaches split data into smaller chunks, encrypt them, and store them across nodes worldwide. This process creates multiple copies of data and enhances resilience against data loss.
Third, it solves the issue of wasted resource consumption in inefficient mining. Bitcoin’s PoW mechanism has long been criticized for its excessive energy consumption, while decentralized storage empowers users to become nodes, earning rewards by contributing idle storage resources. A large number of storage nodes also means lower costs, suggesting that decentralized cloud storage could potentially capture a portion of the Web2 cloud service market share. With continuously improving network bandwidth and hardware capabilities, this represents an enormous market. According to Business Research Insights, the global database market is projected to exceed $120 billion by 2028.
2. Decentralized Storage Architecture
To build truly decentralized applications, decentralized databases should also be part of the Web3 application architecture. These can be divided into four main components: smart contract layer, file storage, database, and general infrastructure layer.
The smart contract layer functions similarly to Layer1, while the general infrastructure layer includes, but is not limited to, oracles, RPCs, access control, identity, off-chain computation, and indexing networks.
Although less perceptible to users, both the file storage and database layers play crucial roles in Web3 application development. They provide essential infrastructure for storing structured and unstructured data, requirements common across various applications. Given the nature of this report, the following sections will focus on these two components.
2.1 Decentralized File Storage Networks (DFSNs)
DFSNs such as Filecoin, Arweave, and Crust are primarily used for persistent storage of unstructured data—data that doesn't follow predefined formats and doesn't require frequent updates or retrieval. As such, DFSNs are typically used to store various static types of data, including text documents, images, audio files, and videos.
A key advantage of storing this type of data in a distributed architecture is leveraging edge storage devices or edge data centers to bring data physically closer to end-users. This approach offers reduced network communication costs, lower interaction latency, and decreased bandwidth overhead. It also provides greater adaptability and scalability. For example, Storj charges just $4.00 per month for 1TB of storage, compared to Amazon S3—the market leader—which charges approximately $23.00 monthly for the same amount of data.
Compared to traditional centralized cloud storage solutions, users benefit from more cost-effective storage options. The decentralized nature of DFSNs also provides higher data security, privacy, and control, since data is distributed across multiple nodes or miners rather than stored on a single centralized server.
2.2 Decentralized Databases
The limitations of storing unstructured files in DFSNs are evident, especially regarding efficient data retrieval and updates. For frequently updated data, these architectures are suboptimal. In such cases, traditional databases like MySQL and Redis remain better choices for developers, having been extensively optimized and tested during the Web2.0 era.
In particular, storing structured data is an unavoidable requirement for applications such as blockchain games and social networks. Traditional databases offer effective ways to manage large volumes of dynamic data and control access to them. They provide indexing, querying, and data manipulation features critical for applications relying on structured data. Therefore, whether based on DFSNs or custom-built underlying storage, high-performance, highly available decentralized databases represent a vital branch within the storage landscape.
3. Technical Analysis of DFSNs
3.1 Overview
Among current Web3 projects, decentralized file storage networks (DFSNs) can be broadly categorized into two types. The first includes projects like Filecoin and Crust, which are built on IPFS implementations. The second includes projects like AR, Sia, and Storj, each with their own underlying protocols or storage systems. Despite differing implementation methods, they all face the same challenge: ensuring genuinely decentralized storage while achieving efficient data storage and retrieval.
Since blockchains themselves are unsuitable for storing large volumes of data on-chain—due to prohibitive costs and impact on block space—ideal decentralized storage networks must be capable of storing, retrieving, and maintaining data while ensuring all participants are properly incentivized and adhere to trust mechanisms inherent in decentralized systems.
We will evaluate several mainstream projects based on the following technical aspects:
Data Storage Format: The storage protocol layer must define how data should be stored—for instance, whether data should be encrypted, and whether it should be stored as a whole or split into small hashed blocks.
Data Replication and Backup: Decisions must be made about where data is stored—e.g., how many nodes should retain copies, whether all data should be replicated across all nodes, or whether each node receives different fragments to further protect data privacy. Data format and dissemination determine data availability probability on the network, i.e., persistence over time despite node failures.
Long-Term Data Availability: The network must ensure data remains accessible over time and location. This involves designing incentive mechanisms to prevent storage nodes from deleting old data over time.
Proof of Stored Data: The network needs not only to know where data is stored, but storage nodes must also prove they actually store the data they claim to, determining their share of incentives.
Storage Pricing Mechanism: Nodes are expected to pay for ongoing file storage.
3.2 Data Storage and Replication
As previously mentioned, Filecoin and Crust use IPFS as the network protocol and communication layer for transferring files between peers and storing them on nodes. The difference lies in Filecoin using erasure coding (EC) to achieve scalable data storage. Erasure coding (EC) is a data protection method that divides data into segments, expands and encodes redundant data blocks, and stores them in different locations—such as disks, storage nodes, or geographic regions. EC creates a mathematical function describing a set of numbers, allowing accuracy verification and recovery if one number is lost.
Source: usenix
The basic equation is n=k+m, where total data blocks equal original data blocks plus parity blocks.
From k original data blocks, m parity blocks are calculated. Storing these k+m blocks across k+m disks tolerates failure of any m disks. When a disk fails, the original data can be reconstructed using any k surviving blocks. Similarly, distributing k+m blocks across different storage nodes allows tolerance of up to m node failures.
When new data is to be stored on the Filecoin network, users must connect via the Filecoin Storage Market to a storage provider and negotiate terms before placing a storage order. Users must decide which type of erasure code and replication factor to use. Through erasure coding, data is broken into fixed-size segments, expanded, and redundantly encoded so that only a subset of segments is needed to reconstruct the original file. The replication factor refers to how many times data is copied across additional storage sectors of storage miners. Once the miner and user agree on terms, data is transferred to the miner and stored in their storage sectors.
Crust’s data storage method differs—it replicates data to a fixed number of nodes: upon submitting a storage order, data is encrypted and sent to at least 20 Crust IPFS nodes (adjustable). On each node, data is split into smaller pieces, hashed into a Merkle tree. Each node retains all fragments constituting the complete file.
Arweave also uses full-file replication but employs different techniques. After a transaction is submitted to the Arweave network, a single node stores the data as a block on the Blockweave (Arweave’s blockchain structure). From there, a highly aggressive algorithm called Wildfire ensures rapid data replication across the network—because to mine the next block, any node must prove access to the previous block.
Sia and Storj also use EC for file storage. In fact, Crust’s implementation—storing 20 full datasets across 20 nodes—is extremely redundant, enhancing durability but inefficient in bandwidth usage. Erasure coding offers a more efficient way to achieve redundancy, increasing data persistence without heavy bandwidth impact. Sia and Storj directly distribute EC shards across a specific number of nodes to meet desired persistence levels.
3.3 Proof of Storage and Incentives
We first discuss data storage formats because the chosen technical path directly influences differences in proof and incentive mechanisms across protocols—specifically, how to verify that data intended for a specific node is indeed stored there. Only after verification can other mechanisms ensure data remains stored over time (i.e., storage nodes don’t delete data post-initial storage).
Such mechanisms include algorithms proving data was stored over a period, financial incentives for successfully completing storage requests, and penalties for failed obligations. This section outlines the storage and incentive protocols of each project.
3.3.1 Filecoin
On Filecoin, storage miners must deposit collateral into the network before accepting any storage requests, committing to provide storage. Once done, miners can offer storage on the marketplace and price their services. Filecoin innovatively introduced PoRep and PoSt for miner storage validation.
Source: Filecoin
Proof-of-Replication (PoRep): Miners must prove they store unique copies of data. Unique encoding ensures two storage deals for the same data cannot reuse the same disk space.
Proof-of-Spacetime (PoSt): Throughout the storage deal lifecycle, storage miners must prove every 24 hours that they continue dedicating exclusive storage space to the data.
After submitting proofs, storage providers receive FIL rewards. Failure to uphold commitments results in slashing (confiscation) of staked tokens.
Over time, storage miners must consistently prove ownership through regular algorithm execution. However, such consistent checks require substantial bandwidth. Filecoin’s innovation lies in reducing bandwidth use: miners use the output of the previous proof as input for the current one, sequentially generating replication proofs. This is achieved through multiple iterations representing the duration data must be stored.
3.3.2 Crust Network
Like Filecoin, Crust maintains an incentive-layer-and-storage-layer relationship with IPFS. In Crust Network, nodes must deposit collateral before accepting storage orders. The amount of storage space a node offers determines the maximum collateral amount, which is staked and allows participation in block creation. This algorithm is known as Guaranteed Proof of Stake (GPoS), ensuring only stakeholders in the network can provide storage.
Source: Crust Wiki
Unlike Filecoin, Crust’s pricing mechanism relies on DSM—nodes and users automatically connect to the Decentralized Storage Market (DSM), which automatically selects which nodes will store user data. Prices are determined based on user demands (e.g., storage duration, space, replication factor) and network factors (e.g., congestion). When users submit orders, data is sent to multiple nodes, which use Trusted Execution Environments (TEE) to split and hash fragments. Since TEE is a secure hardware component inaccessible even to owners, node operators cannot reconstruct files independently.
After file storage, work reports containing file hashes are published alongside remaining node storage to the Crust blockchain. To ensure long-term storage, the network periodically requests random data checks: within TEE, random Merkle tree hashes and associated file fragments are retrieved, decrypted, and rehashed. The new hash is then compared to the expected one. This storage proof implementation is called Meaningful Proof of Work (MPoW).
GPoS is a PoS consensus algorithm where stake limits are defined by storage resources. Using workload reports from the first-layer MPoW mechanism, Crust’s chain obtains all nodes’ storage workloads. The second-layer GPoS algorithm calculates a staking quota per node based on workload, which then governs PoS consensus. Block rewards are proportional to each node’s stake, whose upper limit is constrained by the amount of storage offered.
3.3.3 Arweave
Compared to the prior two models, Arweave uses a fundamentally different pricing model: all data stored on Arweave is permanent, with prices based on the cost of storing data for 200 years.
Arweave’s underlying data network is based on the Blockweave block generation model. Typical blockchains like Bitcoin use a linear chain, where each block links to the previous one. In contrast, Blockweave uses a mesh structure where each block links not only to the previous block but also to a randomly selected past “recall block.” Recall blocks are deterministically but unpredictably determined by the hash and height of the previous block in history. To mine or validate a new block, miners must access information from the recalled block.
Arweave’s PoA uses the RandomX hashing algorithm. Miner block probability = recall block access probability × chance of finding the correct hash first. Miners use PoW to find valid hashes for new blocks, but the nonce depends on both the previous block and a random recall block. The randomness encourages miners to store more blocks to increase computational success and block reward chances. PoA also incentivizes miners to store “scarce blocks”—those others haven’t stored—to gain higher block probabilities and rewards.
Source: Arweave Yellow Paper
With one-time fees and free subsequent reads, sustainability means users can always access data—but how do you long-term incentivize miners to provide zero-revenue read services?
Source: Arweave Yellow Paper
Inspired by BitTorrent’s game-theoretic strategy “optimistic tit-for-tat algorithm,” where nodes cooperate optimistically and punish non-cooperation, Arweave designed Wildfire—a hidden incentive-based node scoring system. Nodes score neighboring peers based on data received and response speed, prioritizing higher-ranked peers for requests. Higher-ranked nodes enjoy greater credibility, higher block probabilities, and better chances of accessing scarce blocks.
Wildfire is essentially a game—highly scalable and adaptive. There is no consensus on rankings among nodes, nor obligation to disclose ranking logic. Node reputations (“good” vs. “bad”) self-adjust via adaptive mechanisms that determine rewards or penalties for new behaviors.
3.3.4 Sia
Like Filecoin and Crust, storage nodes must deposit collateral to provide storage. On Sia, nodes decide how much collateral to post—directly impacting user pricing, but low deposits mean little penalty if nodes disappear from the network. These forces push nodes toward balanced collateralization.
Users connect to storage nodes via an automated market similar to Filecoin: nodes set prices, users set target and expected prices based on desired storage duration. Then users and nodes automatically pair.
Source: Crypto Exchange
Among these projects, Sia uses the simplest consensus: storing contracts on-chain. After agreement, funds are locked in a contract, data is split via erasure coding into segments, each separately hashed with different encryption keys, then replicated across several nodes. The Sia blockchain records the contract terms and Merkle tree hash list of the data. To ensure storage over the agreed period, periodic storage proofs are submitted. These proofs are based on random parts of the original file and the Merkle tree hash list recorded on-chain. Each submitted proof earns rewards, with final payment upon contract completion.
On Sia, storage contracts last up to 90 days. For longer storage, users must manually reconnect via Sia client software to renew the contract another 90 days. Skynet, a layer atop Sia (similar to Filecoin’s Web3.Storage or NFT.Storage), automates renewal via its own client instances. While functional, this isn’t a protocol-level solution.
3.3.5 Storj
The Storj decentralized storage network lacks a blockchain or blockchain-like structure. No blockchain means no global state consensus. Instead, satellite nodes track data locations across storage nodes, while storage nodes handle actual storage. Satellites choose which storage nodes to use, and storage nodes decide which satellites to accept requests from.
Beyond tracking data locations, satellites also handle billing and payments for storage and bandwidth usage. In this setup, storage nodes set their own prices, and satellites connect them to users willing to pay.
Source: Storj GitHub
When users want to store data on Storj, they select a satellite node and share their storage needs. The satellite picks suitable storage nodes and connects them to the user. Users then transfer files directly to storage nodes while paying the satellite. The satellite pays storage nodes monthly for stored files and bandwidth used.
This technical design is quite centralized—satellite nodes are fully controlled by the project team, meaning the team holds pricing power. While centralization enables high performance, as noted earlier, distributed storage doesn’t guarantee decentralization. Storj’s ERC-20 token $STORJ on Ethereum doesn’t leverage smart contract functionality; it merely serves as an alternative payment method.
This relates closely to Storj’s business model: targeting enterprise-grade storage, directly competing with Amazon S3, and partnering with Microsoft Azure to deliver performance matching or exceeding AWS. Though performance metrics remain undisclosed, Storj’s significantly lower costs suggest viable business potential for decentralized storage.
4. Impact of Different Technical Approaches
4.1 Economic Models
Technical choices also shape token model design. Each of the four major decentralized storage networks has its own economic model.
Filecoin, Crust, and Sia all use a Stake-for-Access (SFA) token model. Under this model, storage providers must lock native network assets to accept storage deals. The amount locked scales with the volume of data stored. Thus, providers must increase collateral as they store more data, increasing demand for the native asset. Theoretically, asset price should rise with growing stored data volume.
Arweave instead uses a unique donation-based model: a large portion of each one-time storage fee goes into a donation pool. Over time, tokens in the pool accrue interest in storage purchasing power and are gradually distributed to miners, ensuring long-term data persistence. This effectively locks tokens long-term: as storage demand grows, more tokens are removed from circulation.
Compared to the other three, Storj’s token model is the simplest. Its $STORJ token serves solely as payment for storage services—used by both end users and providers, like all other networks. Hence, $STORJ price is a direct function of demand for storage services.
4.2 Target Users
It's difficult to objectively say one storage network is better than another. In designing decentralized storage networks, there is no single optimal solution. Trade-offs must be made across technical design, tokenomics, and community building depending on the network’s purpose and problems it aims to solve.
Filecoin primarily targets enterprises and app developers, offering cold storage solutions. Its competitive pricing and accessibility make it an attractive alternative for Web2 entities seeking cost-effective archival storage.
Crust ensures high redundancy and fast retrieval, making it suitable for high-traffic dApps and efficient retrieval of popular NFT data. However, lack of persistent redundancy severely limits its ability to offer permanent storage.
Arweave stands out with its concept of permanent storage, particularly popular for storing Web3 data such as blockchain state and NFTs. Other networks mainly optimize for hot or cold storage.
Sia targets the hot storage market, focusing on developers seeking fully decentralized, private storage with fast retrieval. While currently lacking native AWS S3 compatibility, access layers like Filebase offer such functionality.
Storj appears more comprehensive but sacrifices some decentralization. It significantly lowers entry barriers for AWS users, catering to enterprise audiences focused on hot storage optimization. It offers Amazon S3-compatible cloud storage.
5. Ecosystem Development in Decentralized Storage
Regarding ecosystem development, we can identify two types: first, upper-layer DApps built entirely on storage networks to enhance functionality and ecosystems; second, existing decentralized apps and protocols like Opensea and AAVE choosing to integrate with specific storage networks to become more decentralized. This section focuses on Filecoin, Arweave, and Crust, as Sia and Storj show less notable ecosystem activity.
5.1 Filecoin Ecosystem
Source: Filecoin
Within Filecoin’s ecosystem, 115 projects fall into the first category—fully built on Filecoin’s underlying infrastructure. Most are concentrated in general storage, NFTs, and consumer storage. Another milestone for Filecoin is the Filecoin Virtual Machine (FVM), analogous to the Ethereum Virtual Machine (EVM), providing an environment to deploy and execute code within smart contracts.
Source: Filecoin
With FVM, the Filecoin network gains smart contract execution capability atop its existing storage network. Developers won’t program users’ storage data directly, but define how that data—once stored trustlessly via smart contracts—can automatically or conditionally operate. Potential scenarios include:
Distributed computing based on data stored on Filecoin (compute at data location without moving it)
Crowdfunded dataset preservation (e.g., anyone can fund storage of socially important data like crime or climate change records)
Smart storage markets (e.g., dynamically adjust rates based on time of day, replication level, regional accessibility)
Centuries-long storage and perpetual custody (e.g., store data usable across generations)
Data DAOs or tokenized datasets (e.g., model data value as tokens and form DAOs to coordinate and trade computations on top)
Locally stored NFTs (e.g., co-locate NFT content with NFT registry records)
Time-locked data retrieval (e.g., unlock datasets only after public disclosure of company records)
Collateralized loans (e.g., lend to storage providers for specific purposes like accepting FIL+ deals from certain users or expanding capacity within a timeframe)
Source: Filecoin
At its core, FVM is WebAssembly (WASM)-based. This allows developers to write native applications in any language compilable to WASM, lowering the barrier for Web3 developers who can use familiar tools and avoid steep learning curves tied to specific languages.
Developers can also port existing Ethereum smart contracts with minimal or no source code changes. Reusing audited, battle-tested Ethereum contracts saves development time and cost, allowing users to benefit from utility with minimal risk.
Another notable initiative is Filecoin Plus—a program subsidizing users to store large, valuable datasets at discounted rates. Clients uploading data can apply to a community-selected group of members called Notaries, who review and allocate DataCaps. Clients then use DataCaps to subsidize transactions with storage providers.
Filecoin Plus brings multiple benefits: increased network activity, sustained demand for blockspace from valuable data storage, improved service quality at competitive prices, and rising block rewards. Compared to 2021, stored data volume increased 18-fold after Filecoin Plus launched in 2022.
5.2 Crust Network Ecosystem
Compared to Filecoin and Arweave, Crust takes a different path in ecosystem development. It leans toward direct collaboration with existing Web3 applications rather than incentivizing third-party developers to build on Crust. This is partly because Crust is built on Polkadot. Although Ethereum and Cosmos were considered early on, their technical paths weren’t sufficiently compatible. Crust prefers Polkadot’s Substrate framework for its highly customizable development space, on-chain upgrades, and governance.
Source: Crust Network
Crust excels in developer support. It introduced the Crust Developer Toolkit—including JS SDK, GitHub Actions, Shell Scripts, and IPFS Scan—to accommodate integration preferences across Web3 projects. Currently, the toolkit is integrated into various Web3 projects such as Uniswap, AAVE, Polkadot Apps, Liquity, XX Messenger, and RMRK.
According to official data, over 150 projects are now integrated with Crust Network. A significant portion (over 34%) are DeFi projects, largely due to their high-performance data retrieval requirements.
As noted, data on Crust is replicated to at least 20 nodes, often over 100. While this demands higher initial bandwidth, simultaneous retrieval from multiple nodes accelerates access and provides robust redundancy during failures or node departures. Crust relies on this high redundancy level because, unlike other chains, it lacks data replenishment or repair mechanisms. Among decentralized storage networks, Crust Network is the youngest.
5.3 Arweave Ecosystem
Source: Arweave, the newest ecosystem landscape
As shown above, Arweave also boasts a strong ecosystem. Around 30 applications are highlighted as being fully built on Arweave. While fewer than Filecoin’s 115, these still cover fundamental user needs across infrastructure, exchanges, social, and NFT domains.
Notably, decentralized databases are being built on Arweave. Arweave primarily uses its block structure for data storage while executing off-chain computation on the user side. Thus, Arweave’s cost depends only on the volume of on-chain stored data.
This separation of computation from the chain, known as the Storage-based Consensus Paradigm (SCP), addresses blockchain scalability challenges. SCP works on Arweave because on-chain data input ensures off-chain computation produces the same verifiable state as on-chain computation.
Successful SCP implementation has opened doors for developing multiple databases on Arweave. Four distinct databases built on Arweave include:
● WeaveDB: A key-value database built as an Arweave smart contract, using whitelisted addresses for access control.
● HollowDB: A key-value database built as an Arweave smart contract, using whitelisted addresses and ZK proofs for data verifiability.
● Kwil: A SQL database running its own P2P node network, using Arweave as the storage layer. It uses public/private key pairs for access control and its own consensus for data validation.
● Glacier: A NoSQL database architected as a ZK-Rollup, using Arweave as its data availability layer. It uses public/private key pairs for access control and ZK proofs for data verifiability.
6. Growth Drivers
Decentralized storage growth depends on several core factors, grouped into three broad categories: overall market outlook, technology, and public awareness. These factors are interrelated and mutually reinforcing, further divisible into nuanced subcategories detailed below.
6.1 Market Outlook
6.1.1 Potential of the Cloud Storage Market
As the internet becomes integral to modern life, cloud storage services are essential for nearly everyone. In 2022, the global cloud storage market reached an impressive $78.6 billion, with no signs of slowing. Market research suggests the industry could reach $183.75 billion by 2027.
Meanwhile, IDC forecasts the cloud storage market to reach $3
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
@gryphsisacademy
