The Current State and Future of Web3 Decentralized Database Storage
TechFlow Selected TechFlow Selected
The Current State and Future of Web3 Decentralized Database Storage
Why do we need centralized databases? How to evaluate the quality of decentralized databases?
Navigating Web3 tides with focused insightsAuthor: Maggie
Why Do We Need Decentralized Databases?
Web2 applications have two fundamental data storage methods: file systems (File System) and databases (Database). Due to the lack of database products in Web3, most DApps still rely on centralized databases to store structured data, aside from storing a small amount of critical data in expensive smart contracts. As decentralized file systems like IPFS are increasingly used to store NFT data for Web3 applications, they have gained recognition and adoption within the Web3 ecosystem. Meanwhile, decentralized database technologies have also undergone a round of iteration, giving rise to various new products.
Decentralized databases offer unique advantages over traditional centralized databases, reducing single points of failure for Web3 projects and enabling DApps to achieve full decentralization.
Decentralized databases are suitable for storing frequently accessed hot data—non-financial data of DApps, such as:
-
NFT metadata
-
DAO voting data
-
DEX order books
-
Decentralized social data, blog content, emails
-
Complex relational database data required by DApps
What Types of Decentralized Database Storage Systems Exist?
In recent years, numerous decentralized database projects have emerged, with some innovative ones gaining significant attention.
-
Ceramic: Launched in 2019, Ceramic stores and manages data as streams, appending formatted event logs to these streams. The logs are turned into files and uploaded to IPFS. It provides GraphQL API queries. Ceramic does not have an incentive model like IPFS but supports create, read, and update operations (CRU).
-
OrbitDB: An earlier project compared to Ceramic, OrbitDB also uses the IPFS file system for storage. It supports NoSQL databases and file storage.
-
Tableland: Launched in 2022 and currently in public test phase, Tableland's production version is expected to launch in 2023. Data storage requires using smart contracts that define SQL statements and set permissions. Reading data occurs off-chain and is free. Currently, the contract has been deployed on Ethereum and L2s such as Optimism.
-
Polybase: Now running on testnet, Polybase is a NoSQL database supporting CRUD operations, each requiring payment. Additionally, Polybase supports various file systems for storing database files, including local disks, IPFS, Filecoin, Polystore, and even AWS S3. It also uses payment channels for query payments, reducing on-chain transaction frequency and avoiding query delays caused by payments.
-
Web3Q: Launched in 2022 with its testnet live, it introduces a novel URL pattern called Web//access protocol for data access. Its pricing model is unique—users can receive refunds when deleting data.
-
Kwil: A SQL database system built on Arweave, using smart contracts for payments.
-
KYVE: A database system based on Arweave.
Technically
-
Both SQL and NoSQL can be used for databases. SQL is more mature and efficient, while NoSQL is richer and more flexible. SQL requires highly consistent data structures and offers stronger join query capabilities, making it mature and efficient. NoSQL’s key-value format better aligns with Ethereum’s design patterns, supports diverse data types, and is flexible and easy to scale.
-
Functionally, supporting CRUD is ideal, but supporting updates and deletes adds complexity to the system. If the system uses local storage, it may not support historical value queries. When using IPFS or Arweave, the database must be append-only; otherwise, multiple versions of one piece of data would double storage costs.
-
There are two options for underlying file systems:
-
Storing files locally offers greater flexibility, customizable retrieval logic, higher efficiency, and avoids the unreliability and complexity associated with decentralized file systems like Arweave. For example: users pay database miners in TokenA, who then need to pay Arweave coin to store data—this two-layer network setup increases complexity.
-
Database files stored in decentralized file systems such as IPFS and Arweave;
-
Stored locally on nodes or on cloud storage like S3.
-
-
Similar to decentralized storage, improving data retrieval speed, designing effective incentive models and token economics, and implementing availability assurance algorithms are key factors determining whether a protocol will gain widespread adoption.
-
A good incentive and token model not only encourages node participation but also incentivizes nodes to act correctly—for instance, providing effective retrieval services instead of merely storing data to earn rewards.
-
Data availability assurance algorithms periodically check nodes’ data storage and require them to provide proofs of data availability. These proofs work in tandem with incentives to prevent data loss.
-
Data retrieval speed significantly impacts user experience and is crucial for the convenience and smooth operation of DApps.
-
Summary
-
The decentralized database space holds high potential and urgent demand, yet no product has achieved broad acceptance and usage so far.
-
Decentralized database technology is less mature than decentralized file storage systems because it builds upon distributed file systems. Many projects started in 2022.
-
Improving data retrieval speed, incentive models, token economics, and data availability assurance algorithms are critical determinants of a protocol’s widespread adoption. The focus will be on reducing retrieval time, which is essential for DApp usability and performance.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
Foresight Ventures
