Decentralized Data Layer: The New Infrastructure for the AI Era
TechFlow Selected TechFlow Selected
Decentralized Data Layer: The New Infrastructure for the AI Era
When focusing on the vertical domain of data resources, emerging Web projects offer new possibilities for data acquisition, sharing, and utilization.
Navigating Web3 tides with focused insightsAuthor: IOSG Ventures
TL/DR
We’ve previously discussed how AI and Web3 can complement each other across verticals such as computing networks, agent platforms, and consumer applications. Focusing specifically on the data resources vertical, emerging Web3 projects are unlocking new possibilities for data access, sharing, and utilization.
-
Traditional data providers struggle to meet the demands of AI and other data-driven industries for high-quality, real-time, verifiable data—particularly in terms of transparency, user control, and privacy protection.
-
Web3 solutions are reshaping the data ecosystem. Technologies like MPC, zero-knowledge proofs, and TLS Notary ensure data authenticity and privacy when flowing across multiple sources, while decentralized storage and edge computing offer greater flexibility and efficiency for real-time data processing.
-
A new infrastructure layer—decentralized data networks—has given rise to representative projects such as OpenLayer (a modular layer for authentic data), Grass (leveraging users’ idle bandwidth through a decentralized crawler node network), and Vana (a Layer 1 network for user data sovereignty), each pioneering distinct technical approaches to unlock new frontiers for AI training and applications.
-
Through crowd-sourced capacity, trustless abstraction layers, and token-based incentives, decentralized data infrastructure offers solutions that are more private, secure, efficient, and cost-effective than those provided by Web2 hyperscalers. It also empowers users with control over their data and its derived value, fostering a more open, secure, and interoperable digital ecosystem.
1. The Surge in Data Demand
Data has become a critical driver of innovation and decision-making across industries. UBS predicts that global data volume will grow more than tenfold between 2020 and 2030, reaching 660 ZB. By 2025, each person on Earth is expected to generate 463 EB (exabytes, where 1 EB = 1 billion GB) of data daily. The Data-as-a-Service (DaaS) market is rapidly expanding: according to Grand View Research, the global DaaS market was valued at $14.36 billion in 2023 and is projected to grow at a CAGR of 28.1% through 2030, reaching $76.8 billion. Behind these soaring figures lies a growing demand from multiple sectors for high-quality, real-time, trustworthy data.
AI model training relies heavily on large volumes of input data to identify patterns and adjust parameters. After training, datasets are needed to test model performance and generalization capabilities. Additionally, AI agents—as a foreseeable form of next-generation intelligent applications—require reliable, real-time data sources to ensure accurate decision-making and task execution.
(Source: Leewayhertz)
The demand for business analytics is also becoming more diverse and widespread, serving as a core tool for enterprise innovation. For example, social media platforms and market research firms require reliable user behavior data to formulate strategies and uncover trends, integrating multi-platform data to build more comprehensive user profiles.
Within the Web3 ecosystem, reliable off-chain data is essential to support novel financial products. As more assets are tokenized, flexible and reliable data interfaces are required to enable product innovation and risk management, allowing smart contracts to execute based on verifiable, real-time data.
Beyond these use cases, data demand is surging in fields such as scientific research and the Internet of Things (IoT). These emerging applications highlight an industry-wide need for diverse, authentic, and real-time data—an increasing demand that traditional systems may struggle to meet amid rapid data growth and evolving requirements.
2. Limitations and Challenges of Traditional Data Ecosystems
A typical data ecosystem includes data collection, storage, processing, analysis, and application. Centralized models are characterized by concentrated data collection and storage, managed and maintained by core enterprise IT teams, with strict access controls.
For instance, Google’s data ecosystem spans multiple sources—from its search engine and Gmail to the Android operating system—collecting user data through these platforms, storing it in globally distributed data centers, and using algorithms to process and analyze the data to optimize products and services.
In finance, LSEG (formerly Refinitiv) collects real-time and historical data from global exchanges, banks, and major financial institutions, while leveraging its Reuters News network to gather market-related news. It applies proprietary algorithms and models to generate analytical insights and risk assessments as value-added products.
(Source: kdnuggets.com)
While effective in delivering specialized services, centralized architectures increasingly reveal their limitations—especially in coverage of emerging data sources, transparency, and user privacy. Key challenges include:
-
Incomplete Data Coverage: Traditional providers face difficulties in rapidly capturing and analyzing emerging data sources such as social media sentiment and IoT device data. Centralized systems struggle to efficiently collect and integrate "long-tail" data from numerous small-scale or non-mainstream sources.
The 2021 GameStop event exposed this limitation: investor sentiment on Reddit rapidly shifted market dynamics, but data terminals like Bloomberg and Reuters failed to capture these shifts in time, resulting in delayed market forecasts.
-
Limited Data Accessibility: Monopolies restrict access. While many traditional providers offer partial data via APIs or cloud services, high access costs and complex authorization processes still hinder data integration.
-
On-chain developers face difficulty accessing reliable off-chain data quickly, as high-quality data remains concentrated among a few tech giants with prohibitively high access costs.
-
Lack of Transparency and Trustworthiness: Many centralized data providers lack transparency in their data collection and processing methods, and have no effective mechanisms to verify the authenticity and integrity of large-scale, real-time data. The centralized nature also increases risks of data tampering or manipulation.
-
Privacy Protection and Data Ownership: Large tech companies commercialize user data at scale. As creators of personal data, users rarely receive fair compensation. They often lack visibility into how their data is collected, processed, and used, and have little control over its scope and usage. Excessive data collection poses serious privacy risks.
-
The Facebook-Cambridge Analytica scandal revealed significant gaps in transparency and privacy protection within traditional data ecosystems.
-
Data Silos: Real-time data from disparate sources and formats is difficult to integrate, limiting comprehensive analysis. Much data remains locked within organizations, restricting cross-industry and cross-organizational data sharing and innovation. This silo effect impedes cross-domain data integration.
-
For example, consumer brands need to consolidate data from e-commerce platforms, physical stores, social media, and market research—but inconsistencies in format or platform isolation make integration challenging. Similarly, ride-sharing companies like Uber and Lyft collect vast amounts of real-time data on traffic, passenger demand, and location, but due to competition, cannot share or aggregate this data.
Beyond these issues, concerns around cost efficiency and flexibility persist. While traditional data providers are actively addressing these challenges, emerging Web3 technologies offer new paradigms and possibilities for transformative solutions.
3. Web3 Data Ecosystem
Since the launch of decentralized storage solutions like IPFS (InterPlanetary File System) in 2014, a wave of innovative projects has emerged to address the shortcomings of traditional data ecosystems. Today, decentralized data solutions have evolved into a multi-layered, interconnected ecosystem covering all stages of the data lifecycle—including data generation, storage, exchange, processing and analysis, verification and security, and privacy and ownership.
-
Data Storage: The rapid growth of Filecoin and Arweave demonstrates that decentralized storage (DCS) is becoming a paradigm shift in data storage. DCS reduces single points of failure through distributed architecture and offers competitive cost efficiency, attracting broad participation. With a surge in scalable use cases, DCS storage capacity has grown exponentially (e.g., Filecoin's total storage reached 22 exabytes in 2024).
-
Processing and Analytics: Decentralized computing platforms like Fluence leverage edge computing to enhance real-time data processing, particularly beneficial for latency-sensitive applications such as IoT and AI inference. Web3 projects employ federated learning, differential privacy, trusted execution environments (TEEs), and fully homomorphic encryption to provide flexible privacy-preserving computation.
-
Data Markets / Exchange Platforms: To facilitate data valuation and circulation, Ocean Protocol uses tokenization and DEX mechanisms to create open, efficient data marketplaces—for example, partnering with Daimler (Mercedes' parent company) to develop a data exchange for supply chain optimization. Meanwhile, Streamr has built a permissionless, subscription-based data streaming network ideal for IoT and real-time analytics, showing strong potential in transportation and logistics (e.g., collaboration with Finnish smart city initiatives).
As data exchange and utilization intensify, ensuring data authenticity, trustworthiness, and privacy has become paramount. This has driven innovation in the Web3 ecosystem toward advanced data verification and privacy-preserving solutions.
3.1 Innovations in Data Verification and Privacy Protection
Many Web3-native technologies and projects are tackling the dual challenges of data authenticity and private data protection. Beyond ZK and MPC, Transport Layer Security (TLS) Notary has emerged as a particularly promising verification method.
Introduction to TLS Notary
TLS (Transport Layer Security) is a widely adopted cryptographic protocol designed to secure data transmission between clients and servers, ensuring confidentiality, integrity, and authenticity. It underpins HTTPS, email, instant messaging, and other secure communications.
(TLS Encryption Principle, Source: TechTarget)
Originally conceived a decade ago, TLS Notary introduces a third-party “notary” alongside the client (prover) and server to verify the authenticity of TLS sessions.
Using key-splitting techniques, the TLS session master key is divided between the client and the notary. This design allows the notary to participate in verification without accessing the actual communication content. The mechanism aims to detect man-in-the-middle attacks, prevent fraudulent certificates, ensure data integrity during transit, and allow trusted third parties to confirm session legitimacy—all while preserving communication privacy.
Thus, TLS Notary enables secure data verification while effectively balancing verification needs with privacy preservation.
In 2022, the Ethereum Foundation’s Privacy and Scaling Exploration (PSE) research lab rebuilt the TLS Notary project. The new version, rewritten in Rust, integrates advanced cryptographic protocols such as MPC. It enables users to prove to third parties the authenticity of data received from servers—without revealing the data itself. While retaining the core verification capabilities of the original TLS Notary, the upgraded protocol significantly enhances privacy protections, making it better suited for modern and future data privacy requirements.
3.2 Variants and Extensions of TLS Notary
In recent years, TLS Notary has continued to evolve, spawning several variants that further enhance privacy and verification capabilities:
-
zkTLS: A privacy-enhanced version combining ZKP technology, allowing users to generate encrypted proofs of web data without exposing sensitive information. Ideal for high-privacy communication scenarios.
-
3P-TLS (Three-Party TLS): Introduces a three-party model—client, server, and auditor—enabling the auditor to verify communication security without accessing content. Useful in compliance audits and financial transaction verification where transparency and privacy are both required.
Web3 projects leverage these cryptographic tools to break data monopolies, overcome data silos, and enable trusted data transmission—allowing users to prove ownership of social media accounts, shopping histories for lending, bank records, employment backgrounds, and academic credentials—without compromising privacy. Examples include:
-
Reclaim Protocol uses zkTLS to generate zero-knowledge proofs of HTTPS traffic, enabling secure import of activity, reputation, and identity data from external websites without exposing sensitive details.
-
zkPass leverages 3P-TLS to allow users to verify real-world private data without disclosure, applicable to KYC and credit services, and compatible with the existing HTTPS web infrastructure.
-
Opacity Network, based on zkTLS, enables users to securely prove their activities across platforms (e.g., Uber, Spotify, Netflix) without direct API access, enabling cross-platform activity verification.
(Projects working on TLS Oracles, Source: Bastian Wetzel)
As a crucial link in the data ecosystem, Web3 data verification holds vast application potential, driving a more open, dynamic, and user-centric digital economy. However, advancements in authenticity verification are just the beginning of building next-generation data infrastructure.
4. Decentralized Data Networks
Some projects combine the above verification technologies to explore deeper innovations at the upstream of the data ecosystem—data provenance, distributed data collection, and trusted transmission. Below we examine three representative projects: OpenLayer, Grass, and Vana, each showcasing unique potential in building next-gen data infrastructure.
4.1 OpenLayer
OpenLayer is one of the a16z Crypto 2024 Spring Crypto Accelerator projects and the first modular authentic data layer, aiming to provide an innovative modular solution for coordinating data collection, verification, and transformation to serve both Web2 and Web3 companies. OpenLayer has attracted support from prominent investors including Geometry Ventures and LongHash Ventures.
Traditional data layers face multiple challenges: lack of trusted verification, limited accessibility due to centralization, poor interoperability and liquidity across systems, and no fair mechanism for data value distribution.
A more tangible issue is the growing scarcity of AI training data. Many websites now deploy anti-bot measures to block large-scale data scraping by AI companies.
In the case of private, proprietary data, the situation is even more complex. Valuable data is often stored privately due to sensitivity, and lacks effective incentive mechanisms. In this context, users cannot safely monetize their private data and thus have little motivation to share it.
To address these issues, OpenLayer combines data verification technologies to build a Modular Authentic Data Layer (MADL), using decentralization and economic incentives to coordinate data collection, validation, and transformation—offering Web2 and Web3 companies a more secure, efficient, and flexible data infrastructure.
4.1.1 Core Components of OpenLayer’s Modular Design
OpenLayer provides a modular platform to streamline data collection, trustless verification, and transformation:
a) OpenNodes
OpenNodes are the core component responsible for decentralized data collection within the OpenLayer ecosystem. Users contribute data via mobile apps or browser extensions. Different operators/nodes can optimize rewards based on their hardware specifications.
OpenNodes support three main data types to meet diverse task needs:
-
Publicly available internet data (e.g., financial, weather, sports, and social media streams)
-
User private data (e.g., Netflix watch history, Amazon order records)
-
Self-reported data from secure sources (e.g., signed by proprietary owners or verified via trusted hardware)
Developers can easily add new data types, specify sources, requirements, and retrieval methods. Users can opt to share de-identified data in exchange for rewards. This design enables continuous expansion to meet evolving data needs, lowers barriers to contribution, and supports diverse use cases.
b) OpenValidators
OpenValidators handle post-collection data verification, enabling consumers to confirm that user-provided data exactly matches the source. All verification methods produce cryptographically provable evidence, and results are auditable. Multiple providers offer each type of proof, allowing developers to choose based on their needs.
In initial use cases—especially for public or private data from internet APIs—OpenLayer uses TLSNotary as its verification solution, exporting data from any web app and proving its authenticity without compromising privacy.
Thanks to its modular design, the verification system can integrate other methods to suit different data types and verification needs, including:
-
Attested TLS connections: Using Trusted Execution Environments (TEEs) to establish authenticated TLS connections, ensuring data integrity and authenticity during transit.
-
Secure Enclaves: Leveraging hardware-level secure environments (e.g., Intel SGX) to process and verify sensitive data with enhanced protection.
-
ZK Proof Generators: Integrating ZKPs to verify data properties or computation results without revealing raw data.
c) OpenConnect
OpenConnect is the core module responsible for data transformation and usability. It processes data from various sources to ensure interoperability across systems and meet application-specific needs. For example:
-
Converting data into oracle formats for direct use by smart contracts.
-
Transforming unstructured raw data into structured formats for AI training preprocessing.
For data from users’ private accounts, OpenConnect provides data anonymization and security components to reduce leakage and misuse risks. To meet real-time data needs for AI and blockchain applications, OpenConnect supports efficient real-time transformation.
Currently, through integration with EigenLayer, OpenLayer AVS operators monitor data requests, scrape and validate data, then report results back to the system. Operators stake or re-stake assets via EigenLayer as economic collateral. Malicious behavior leads to slashing of staked assets. As one of the earliest AVSs (Actively Validated Services) on the EigenLayer mainnet, OpenLayer has already attracted over 50 operators and $4 billion in re-staked assets.
In summary, OpenLayer’s decentralized data layer expands the scope and diversity of usable data without sacrificing utility or efficiency. Through cryptographic techniques and economic incentives, it ensures data authenticity and integrity. Its technology has broad practical applications for Web3 dApps seeking off-chain information, AI models requiring real inputs for training and inference, and companies aiming to segment and target users based on existing identities and reputations. It also enables users to monetize their private data.
4.2 Grass
Grass is the flagship project developed by Wynd Network, aiming to build a decentralized web crawler and AI training data platform. In late 2023, Grass raised a $3.5 million seed round led by Polychain Capital and Tribe Capital. In September 2024, the project secured a Series A round led by HackVC, with participation from notable investors including Polychain, Delphi, Lattice, and Brevan Howard.
We noted earlier that AI training requires new data exposure—one solution being the use of multiple IPs to bypass access restrictions. Grass builds on this idea, creating a distributed crawler node network that leverages users’ idle bandwidth to collect and provide verifiable datasets for AI training. Nodes route web requests through users’ internet connections, access public websites, and compile structured datasets. Edge computing is used for preliminary data cleaning and formatting to improve quality.
Grass adopts a Solana Layer 2 Data Rollup architecture, built atop Solana for higher processing efficiency. Validators receive, verify, and batch web transactions from nodes, generating ZK proofs to ensure data authenticity. Verified data is stored in a data ledger (L2) and linked to corresponding L1 chain proofs.
4.2.1 Key Components of Grass
a) Grass Nodes
Similar to OpenNodes, end-users install the Grass app or browser extension and run it, contributing idle bandwidth for web crawling. Nodes route web requests, access public sites, and compile structured datasets, using edge computing for initial cleanup and formatting. Users earn GRASS tokens based on contributed bandwidth and data volume.
b) Routers
Routers connect Grass nodes to validators, managing the node network and relaying bandwidth. Routers are incentivized and rewarded proportionally to the total validated bandwidth they relay.
c) Validators
Validators receive, verify, and batch web transactions from routers, generating ZK proofs. They use unique key sets to establish TLS connections and select appropriate cipher suites for communication with target servers. Grass currently uses centralized validators but plans to transition to a validator committee.
d) ZK Processor
Receives session data proofs from validators, batches validity proofs for all web requests, and submits them to Layer 1 (Solana).
e) Grass Data Ledger (Grass L2)
Stores complete datasets and links them to corresponding L1 (Solana) proofs.
f) Edge Embedding Models
Responsible for converting unstructured web data into structured models suitable for AI training.
Source: Grass
Comparative Analysis: Grass vs. OpenLayer
Both OpenLayer and Grass leverage distributed networks to give companies access to open internet data and authenticated closed information. Both use incentive mechanisms to promote data sharing and high-quality data production. While both aim to build a decentralized data layer to solve data access and verification issues, they differ slightly in technical approach and business model.
Differences in Technical Architecture
Grass uses a Layer 2 Data Rollup architecture on Solana, currently relying on a centralized validator. OpenLayer, as one of the first AVSs on EigenLayer, leverages economic incentives and slashing mechanisms to achieve decentralized validation. Its modular design emphasizes scalability and flexibility in data verification services.
Product Differences
Both offer similar consumer-facing products allowing users to monetize data via nodes. On the B2B side, Grass offers a compelling data marketplace model, using L2 to store full datasets verifiably, providing AI companies with structured, high-quality, verifiable training sets. OpenLayer does not currently have a dedicated data storage component but offers broader real-time data stream verification services (Vaas). Beyond AI data, it serves use cases requiring fast response times—such as price feeds for RWA/DeFi/prediction markets or real-time social data.
Thus, Grass currently targets AI companies and data scientists needing large-scale, structured training datasets, as well as research institutions and enterprises requiring extensive web data. OpenLayer primarily serves on-chain developers needing off-chain data, AI companies requiring real-time, verifiable data streams, and Web2 companies supporting innovative user acquisition strategies—such as verifying competitor usage history.
Future Competitive Dynamics
However, industry trends suggest functional convergence may occur. Grass may soon offer real-time structured data. As a modular platform, OpenLayer could expand into dataset management with its own data ledger. Thus, their competitive domains may gradually overlap.
Both projects may also incorporate data labeling—a critical step. Grass may move faster here, given its massive node network—reportedly over 2.2 million active nodes. This advantage positions Grass to potentially offer Reinforcement Learning from Human Feedback (RLHF) services, leveraging large volumes of labeled data to optimize AI models.
Yet, OpenLayer’s expertise in data verification and real-time processing, combined with its focus on private data, may preserve its edge in data quality and trustworthiness. Moreover, as an EigenLayer AVS, OpenLayer may advance further in decentralized validation mechanisms.
While competition may arise in certain areas, each project’s unique strengths and technical paths may allow them to occupy distinct niches within the data ecosystem.
(Source: IOSG, David)
4.3 Vana
Vana is a user-centric data pool network focused on providing high-quality data for AI and related applications. Compared to OpenLayer and Grass, Vana takes a distinctly different technical and business approach. In September 2024, Vana raised $5 million in a round led by Coinbase Ventures, following an $18 million Series A led by Paradigm. Other notable investors include Polychain and Casey Caruso.
Initially launched in 2018 as an MIT research project, Vana aims to be a Layer 1 blockchain specifically designed for user private data. Its innovations in data ownership and value distribution enable users to profit from AI models trained on their data. At its core, Vana enables the circulation and monetization of private data through trustless, private, and attributable Data Liquidity Pools (DLPs) and an innovative Proof of Contribution mechanism:
4.3.1 Data Liquidity Pools (DLPs)
Vana introduces the concept of Data Liquidity Pools (DLPs)—each DLP is an independent peer-to-peer network aggregating specific types of data assets. Users upload private data (e.g., purchase history, browsing habits, social media activity) to a specific DLP and can selectively authorize third-party access. Data is aggregated and managed through these pools, anonymized to protect privacy while enabling commercial use—such as AI model training or market research.
Users submitting data to a DLP receive corresponding DLP tokens (each DLP has its own token). These tokens represent contribution, grant governance rights, and entitle holders to future revenue shares. Unlike one-time data sales, Vana enables ongoing economic participation—users earn recurring income from subsequent data usage, with transparent tracking.
4.3.2 Proof of Contribution Mechanism
Another core innovation is Vana’s Proof of Contribution (PoC) mechanism—the key to ensuring data quality. Each DLP can customize its own contribution function to verify data authenticity and integrity, and assess its impact on AI model performance. Similar to Proof of Work in crypto, PoC allocates rewards based on data quality, quantity, and usage frequency. Smart contracts automatically execute payouts, ensuring contributors are fairly compensated.
Vana’s Technical Architecture
-
Data Liquidity Layer
This is Vana’s core layer, handling data contribution, validation, and recording into DLPs, bringing data onto-chain as transferable digital assets. DLP creators deploy smart contracts defining purpose, validation methods, and contribution parameters. Contributors and custodians submit data for validation; the PoC module evaluates and scores contributions, granting governance and rewards.
-
Data Portability Layer
An open data platform for contributors and developers—the application layer of Vana. It provides a collaborative space to build applications using accumulated data liquidity from DLPs, offering infrastructure for user-owned model training and AI dApp development.
-
Connectome
A decentralized ledger and real-time data flow graph spanning the entire Vana ecosystem. Using Proof of Stake consensus, it records real-time data transactions within Vana, ensures valid DLP token transfers, and enables cross-DLP data access. EVM-compatible, it interoperates with other networks, protocols, and DeFi applications.
(Source: Vana)
Vana offers a distinct path—focusing on data liquidity and value empowerment. This decentralized data exchange model suits not only AI training and data markets but also provides a new solution for cross-platform data interoperability and authorization in the Web3 ecosystem. Ultimately, it aims to create an open internet ecosystem where users own, manage, and benefit from their data—and the intelligent products built upon it.
5. Value Proposition of Decentralized Data Networks
In 2006, data scientist Clive Humby famously stated, “Data is the new oil.” Over the past two decades, we’ve witnessed rapid advancements in “refining” technologies. Big data analytics, machine learning, and other innovations have unlocked unprecedented value from data. According to IDC, the global datasphere will grow to 163 ZB by 2025, most of which will originate from individual users. With the proliferation of IoT, wearables, AI, and personalized services, much of the commercially valuable data of the future will come from individuals.
Pain Points of Traditional Models: Web3’s Unlocking Innovation
Web3 data solutions overcome the limitations of traditional infrastructure through distributed node networks, enabling broader, more efficient data collection and improving real-time access and verification credibility. These technologies ensure data authenticity and integrity while protecting user privacy—enabling a fairer model of data utilization. This decentralized architecture drives the democratization of data access.
Whether through the user-node models of OpenLayer and Grass, or Vana’s monetization of private user data, Web3 not only improves data collection efficiency but also allows ordinary users to share in the economic benefits of data—creating a win-win model where users gain control and value from their data and its derivatives.
Through token economics, Web3 redefines incentive models, establishing a fairer data value distribution system. This attracts vast user participation, hardware resources, and capital, optimizing the operation of the entire data network.
Compared to traditional solutions, they also offer modularity and scalability: OpenLayer’s modular design, for example, enables flexibility for future technological upgrades and ecosystem expansion. These technical advantages optimize data acquisition for AI model training, delivering richer, more diverse datasets.
From data generation and storage to verification, exchange, and analysis, Web3-powered solutions tackle the shortcomings of traditional infrastructure with unique technological advantages—while empowering users to monetize their personal data and fundamentally transform the data economy. As technology evolves and use cases expand, decentralized data layers—alongside other Web3 data solutions—are poised to become foundational infrastructure for the next generation, supporting a wide range of data-driven industries.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
IOSG Ventures
