From Isolation to Collaboration: The Significance of Web3-Native Data Pipelines
TechFlow Selected TechFlow Selected
From Isolation to Collaboration: The Significance of Web3-Native Data Pipelines
In the Web3 market, building a data pipeline can play a key role not only in its decentralized characteristics but also as a practical starting point to capture these opportunities.
Navigating Web3 tides with focused insightsWritten by: Jay : : FP
Translated by: TechFlow
The release of the Bitcoin whitepaper in 2008 triggered a rethinking of the concept of trust. Blockchain subsequently expanded its definition to include the idea of trustless systems and rapidly evolved, asserting that values such as individual sovereignty, financial democratization, and ownership could be applied across existing systems. Of course, before blockchain can be practically implemented, extensive validation and discussion may be required, as its characteristics might appear somewhat radical compared to various existing systems. However, if we remain optimistic about these scenarios, building data pipelines and analyzing valuable information contained within blockchain storage has the potential to become another pivotal turning point for industry development, as we may observe Web3-native business intelligence previously unseen.
This article explores the potential of Web3-native data pipelines by projecting commonly used data pipeline architectures from the existing IT market onto the Web3 environment. It discusses the benefits of these pipelines, challenges that need to be addressed, and their impact on the industry.
1. Singularities Arise from Information Innovation
“Language is one of the most important distinctions between humans and lower animals. It is not merely the ability to produce sounds, but the linking of definite sounds with definite ideas, and using those sounds as symbols for the communication of thought.”
— Darwin
Throughout history, major advances in human civilization have coincided with innovations in information sharing. Our ancestors used language—both spoken and written—to communicate with each other and pass knowledge down to future generations, giving them a significant advantage over other species. The invention of writing, paper, and printing made widespread information sharing possible, leading to major advancements in science, technology, and culture. In particular, Gutenberg’s metal movable type printing of the Bible was a watershed moment, enabling mass production of books and printed materials. This had profound impacts on the Reformation, democratic revolutions, and the dawn of scientific progress.
The rapid development of IT technologies in the 2000s enabled us to gain deeper insights into human behavior. This led to changes in lifestyle, with most modern individuals making various decisions based on digital information. For this reason, we refer to contemporary society as the “era of IT innovation.”
Just two decades after the full commercialization of the internet, AI technology has once again amazed the world. Numerous applications capable of replacing human labor have emerged, and many are discussing how AI will transform civilization. Some people are even in denial, wondering how such a technology could emerge so quickly as to shake the foundations of our society. Despite Moore's Law indicating exponential growth in semiconductor performance over time, the changes brought by GPT have come too suddenly to immediately comprehend.
Interestingly, the GPT model itself is not actually a highly breakthrough architecture. Instead, the AI industry identifies the following as key success factors for GPT models: 1) defining business domains applicable to large user groups, and 2) model fine-tuning via data pipelines—from data collection to final outputs and feedback based on results. In short, by refining service objectives and upgrading data/information processing workflows, these applications achieve innovation.
2. Data-Driven Decision Making Is Everywhere
Most innovations we speak of are actually based on processing accumulated data rather than chance or intuition. As the saying goes, “In capitalist markets, it is not the strongest that survive, but those who adapt best.” Today’s businesses operate in fiercely competitive and saturated markets. Therefore, companies are collecting and analyzing diverse data to capture even the smallest niches.
We may be overly obsessed with Schumpeter’s theory of “creative destruction,” placing excessive emphasis on intuitive decision-making. Yet even exceptional intuition is ultimately the product of an individual’s accumulated data and information. The digital world will penetrate deeper into our lives in the future, with increasing amounts of sensitive information being represented as digital data.
The Web3 market has attracted significant attention due to its potential to give users control over their own data. However, blockchain—the foundational technology behind Web3—is currently more focused on solving the trilemma (security, decentralization, and scalability). For new technologies to be compelling in real-world applications, it is crucial to develop versatile applications and intelligence. We’ve already seen this occur in the big data domain, where since around 2010, methodologies for processing big data and constructing data pipelines have made substantial progress. Within the Web3 context, efforts must be directed toward advancing the industry by establishing data flow systems capable of generating data-driven intelligence.
3. Opportunities Based on On-Chain Data Flows
So, what opportunities can we capture from Web3-native data flow systems, and what challenges must be overcome to seize them?
3.1 Advantages
In short, configuring Web3-native data flows offers value through secure and efficient distribution of reliable data to multiple entities, enabling extraction of valuable insights.
-
Data Redundancy — On-chain data is less likely to be lost and more resilient because protocol networks store data fragments across multiple nodes.
-
Data Security — On-chain data is tamper-proof, having been validated and agreed upon by a decentralized network of nodes.
-
Data Sovereignty — Data sovereignty refers to users’ rights to own and control their own data. With on-chain data flows, users can see how their data is used and choose to share it only with parties who have legitimate access needs.
-
Permissionless and Transparent — On-chain data is transparent and tamper-resistant, ensuring the data being processed is also a reliable source of information.
-
Stable Operation — When data flows are orchestrated by protocols in a distributed environment, the probability of downtime exposure at each level is significantly reduced due to the absence of single points of failure.
3.2 Use Cases
Trust forms the foundation for interactions and decision-making among different entities. Thus, when reliable data can be securely distributed, it means many interactions and decisions can be conducted via Web3 services involving various participants. This helps maximize social capital, allowing us to envision several use cases below.
3.2.1 Service/Protocol Applications
-
Rule-Based Automated Decision Systems — Protocols use key parameters to operate services. These parameters are periodically adjusted to stabilize service conditions and provide optimal user experiences. However, protocols cannot continuously monitor service states and dynamically update parameters in real time. This is where on-chain data flows come into play. They can analyze service status in real time and recommend optimal parameter sets aligned with service requirements (e.g., automatic floating interest rate mechanisms for lending protocols).
-
Credit Market Growth — Traditionally, credit measures an individual’s repayment capacity in financial markets, enhancing market efficiency. However, in the Web3 market, the definition of credit remains unclear due to scarce personal data and lack of inter-industry data governance. Integrating and aggregating information becomes difficult. By building a process that collects and processes fragmented on-chain data, the credit market in Web3 can be redefined (e.g., Spectral’s MACRO [Multi-Asset Credit Risk Oracle] scoring).
-
Decentralized Social / NFT Expansion — Decentralized societies prioritize user control, privacy protection, censorship resistance, and community governance, offering an alternative social paradigm. Pipelines can thus be established to more seamlessly manage and update metadata and facilitate cross-platform migration.
-
Fraud Detection — Web3 services using smart contracts are vulnerable to malicious attacks that may steal funds, compromise systems, and cause de-pegging or liquidity attacks. By creating systems capable of detecting such threats in advance, Web3 services can formulate rapid response plans and protect users from harm.
3.2.2 Collaboration and Governance Initiatives
-
Fully On-Chain DAOs — Decentralized Autonomous Organizations (DAOs) heavily rely on off-chain tools for effective governance and public fund management. By building on-chain data processing workflows, transparent operational processes for DAOs can be created, further enhancing the value of Web3-native DAOs.
-
Mitigating Governance Fatigue — Web3 protocol decisions are typically made through community governance. However, numerous factors may hinder participation, including geographical barriers, surveillance pressure, lack of required expertise, randomly released governance agendas, and poor user experience. If tools can be created to simplify the process from understanding to implementing individual governance proposals, protocol governance frameworks can operate more efficiently and effectively.
-
Open Data Platforms for Collaborative Works — In current academic and industrial sectors, much data and research material are not publicly disclosed, potentially making overall market development highly inefficient. In contrast, on-chain data pools can foster greater collaboration initiatives than existing markets, as they are transparent and accessible to anyone. The development of numerous token standards and DeFi solutions serves as excellent examples. Additionally, public data pools can be operated for various purposes.
3.2.3 Network Diagnostics
-
Index Research — Web3 users create various metrics to analyze and compare protocol statuses. Multiple objective indicators can be studied and displayed in real time (e.g., Nakaflow’s Nakamoto Coefficient).
-
Protocol Metrics — By processing data such as number of active addresses, transaction volume, asset inflows/outflows, and fees generated by the network, protocol performance can be analyzed. This information can assess the impact of specific protocol upgrades, MEV status, and overall network health.
3.3 Challenges
On-chain data possesses unique advantages that can increase industry value. However, fully realizing these advantages requires overcoming many internal and external industry challenges.
-
Lack of Data Governance — Data governance involves establishing consistent and shared policies and standards to facilitate integration of each data primitive. Currently, each on-chain protocol establishes its own standards and retrieves its own data types. The problem lies in the absence of data governance among entities that aggregate protocol data and offer API services to users. This makes service integration difficult, leaving users unable to obtain reliable and comprehensive insights.
-
Inefficient Cost Structure — Storing cold data on-chain can save users costs related to data security and server maintenance. However, storing frequently accessed data requiring intensive computation on-chain may not be cost-effective.
-
Oracle Issues — Smart contracts can only function effectively when they can access real-world data. However, such external data is not always reliable or consistent. Unlike blockchains secured by consensus algorithms, external data lacks determinism. Oracle solutions must continuously evolve to ensure integrity, quality, and scalability of external data without relying on specific application layers.
-
Protocols Are Still Immature — Protocols use their own tokens to incentivize users to maintain services and pay for usage. However, the parameters required to operate protocols (e.g., precise definitions of service users and incentive schemes) are often managed naively. This makes it difficult to verify the economic sustainability of protocols. If many protocols organically connect and form data pipelines, uncertainty about whether the pipeline functions properly increases significantly.
-
Slow Data Retrieval Speed — Protocols typically process transactions via consensus among many nodes, which limits information processing speed and volume compared to traditional IT business logic. This bottleneck is hard to resolve unless all protocols composing the pipeline significantly improve their performance.
-
The Real Value of Web3 Data — Blockchains are isolated systems not yet connected to the real world. When collecting Web3 data, we must consider whether the collected data provides meaningful insights sufficient to justify the cost of building data pipelines.
-
Unfamiliar Syntax — Existing IT data infrastructure and blockchain infrastructure operate very differently. Even the programming languages used differ, with blockchain infrastructure typically using low-level or newly designed languages tailored to blockchain needs. This makes it difficult for new developers and service users to learn how to handle each data primitive, as they must master a new programming language or adopt a new way of thinking about blockchain data processing.
4. Pipelined Web3 Data Lego
Currently, Web3 data primitives are disconnected and independently extract and process data. This makes it difficult to experiment with synergistic effects in information processing. To address this issue, this article introduces common data pipelines used in the IT market and maps existing Web3 data primitives onto this pipeline framework, making use cases more concrete.
4.1 General Data Pipeline Architecture
Building a data pipeline resembles the process of conceptualizing and automating repetitive decision-making in daily life. By doing so, people can access specific-quality information whenever needed and apply it to decision-making. The greater the volume of unstructured data to be processed, the higher the frequency of information usage, or the greater the need for real-time analysis, the more time and cost can be saved in acquiring proactive input for future decisions by automating this series of processes.
The figure above shows a general architecture used in the existing IT infrastructure market for building data pipelines. Data suitable for analysis is collected from appropriate sources and stored in storage solutions suited to the nature of the data and analytical requirements. For example, data lakes provide raw data storage for scalable and flexible analysis, while data warehouses focus on structured data storage optimized for specific business logic queries and analysis. Data is then processed into insights or practical information in various ways.
Each solution layer can also be offered as packaged services. ETL (Extract, Transform, Load) SaaS products that connect the entire process from data extraction to loading are gaining increasing attention (e.g., FiveTran, Panoply, Hivo, Rivery). The sequence is not always unidirectional; depending on organizational needs, layers can interconnect in various configurations. The most critical aspect when building data pipelines is minimizing the risk of data loss as data moves between server layers. This can be achieved by optimizing the degree of server decoupling and using reliable data storage and processing solutions.
4.2 Pipelines in On-Chain Environments
The conceptual diagram of data pipelines introduced earlier can be applied to on-chain environments as shown above. However, note that fully decentralized pipelines cannot currently be formed, as each fundamental component relies to some extent on centralized off-chain solutions. Moreover, the diagram does not yet include all Web3 solutions, and classification boundaries may be blurred—for instance, KYVE functions not only as a streaming platform but also includes data lake capabilities, essentially acting as a data pipeline itself. Similarly, Space and Time is categorized as a decentralized database but offers API gateway services like REST APIs and streaming, as well as ETL services.
4.2.1 Capture/Processing
For ordinary users or dApps to efficiently use or operate services, they need easy identification and access to primary data sources generated within protocols, such as transactions, states, and log events. This layer involves middleware facilitating processes including oracles, messaging, authentication, and API management. Key solutions include:
Streaming/Indexing Platforms
-
Bitquery, Ceramic, KYVE, Lens, Streamr Network, The Graph, block explorers of individual protocols, etc.
Node-as-a-Service and Other RPC/API Services
-
Alchemy, All that Node, Infura, Pocket Network, Quicknode, etc.
Oracles
-
API3, Band Protocol, Chainlink, Nest Protocol, Pyth, Supra Oracles, etc.
4.2.2 Storage
Compared to Web2 storage solutions, Web3 storage solutions offer several advantages such as persistence and decentralization. However, they also have drawbacks, including high costs and difficulties in updating and querying data. Hence, various solutions have emerged to address these shortcomings and enable efficient handling of structured and dynamic data on Web3—each differing in features such as data type handled, whether data is structured, and whether embedded query functionality exists.
Decentralized Storage Networks
-
Arweave, Filecoin, KYVE, Sia, Storj, etc.
Decentralized Databases
-
Arweave-based databases (Glacier, HollowDB, Kwil, WeaveDB), ComposeDB, OrbitDB, Polybase, Space and Time, Tableland, etc.
*Each protocol employs different permanent storage mechanisms. For example, Arweave uses a blockchain-based model similar to Ethereum, permanently storing data on-chain, whereas Filecoin, Sia, and Storj use contract-based models that store data off-chain.
4.2.3 Transformation
In the Web3 context, the transformation layer is just as critical as the storage layer. This is because blockchain architecture fundamentally consists of distributed node sets, making it easier to implement scalable backend logic. In the AI industry, researchers are actively exploring the use of these advantages in federated learning, leading to protocols specifically designed for machine learning and AI operations.
Data Training/Modeling/Computation
-
Akash, Bacalhau, Bittensor, Gensyn, Golem, Together, etc.
*Federated learning is a method for training AI models by distributing raw models across multiple native clients, training them using local data, and then aggregating learned parameters on a central server.
4.2.4 Analysis/Usage
The dashboard services and end-user insight and analytics solutions listed below are platforms that allow users to observe and discover various insights from specific protocols. Some of these solutions also offer API services for final products. However, it should be noted that data from these solutions is not always accurate, as most rely on separate off-chain tools for data storage and processing. Discrepancies among solutions can also be observed.
Meanwhile, there is a platform called “Web3 Functions” that automatically triggers or executes smart contract execution, similar to how centralized platforms like Google Cloud trigger or execute specific business logic. Using this platform, users can implement business logic in a Web3-native manner—not just by extracting insights from on-chain data.
Dashboard Services
-
Dune Analytics, Flipside Crypto, Footprint, Transpose, etc.
End-User Insights and Analytics
-
Chainalysis, Glassnode, Messari, Nansen, The Tie, Token Terminal, etc.
Web3 Functions
-
Chainlink Functions, Gelato Network, etc.
5. Concluding Thoughts
As Kant said, we can only witness the phenomena of things, never touching their essence. Nevertheless, we leverage observational records called “data” to process information and knowledge, witnessing how innovations in information technology drive civilizational advancement. Therefore, building a data pipeline within the Web3 market—beyond its decentralized traits—can play a pivotal role as a starting point for capturing these opportunities in practice. I conclude this article with several reflections.
5.1 The Role of Storage Solutions Will Become More Important
The most essential prerequisite for owning a data pipeline is establishing data and API governance. In an increasingly diverse ecosystem, norms created by each protocol will continue to be recreated, and fragmented transaction records across multi-chain ecosystems will make it harder for individuals to derive comprehensive insights. At this point, “storage solutions” become entities capable of collecting fragmented information, updating per-protocol norms, and providing integrated data in unified formats. We observe that existing market leaders in storage solutions (such as Snowflake and Databricks) are rapidly growing with large customer bases by vertically integrating multiple pipeline layers and leading industry development.
5.2 Opportunities in the Data Source Market
When data becomes more accessible and processing improves, successful use cases begin to emerge. This creates a positive feedback loop where data sources and collection tools proliferate explosively—since 2010, due to major technological advances in data pipeline construction, both the types and volumes of digitally collected data have grown exponentially each year. Applying this context to the Web3 market, many data sources may recursively generate on-chain in the future. This implies blockchain will expand into diverse business domains. At this juncture, we can anticipate advancements in data acquisition through data markets like Ocean Protocol, DeWi (Decentralized Wireless) solutions like Helium and XNET, and storage solutions.
5.3 Meaningful Data and Analysis Matter Most
However, most importantly, we must continuously ask what data should be prepared to extract truly needed insights. Nothing is more wasteful than building data pipelines without clear hypotheses to validate. While the existing market has achieved numerous innovations through data pipelines, it has also paid countless costs through repeated meaningless failures. Constructive discussions about tech stack development are valuable, but the industry needs time to reflect on more fundamental questions—such as what data should be stored in blockspace or what purpose the data should serve. The “goal” should be realizing Web3’s value through actionable intelligence and use cases, while developing multiple primitives and completing the pipeline is merely the “means” to achieve that goal.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
Four Pillars
