Let Data Flow: How Crypto Projects Are Alleviating the AI Data Training Bottleneck?
TechFlow Selected TechFlow Selected
Let Data Flow: How Crypto Projects Are Alleviating the AI Data Training Bottleneck?
In the race to create the most intelligent and human-like AI models, the key resource is data.
Navigating Web3 tides with focused insightsAuthor: SHLOK KHEMANI
Translation: TechFlow
See if you can spot all the carefully curated references in the image
Over the past two years, since a relatively unknown startup called OpenAI released a chatbot application named ChatGPT, AI has moved from behind the scenes to center stage. We are at an inflection point where machine intelligence is permeating every aspect of our lives. As competition intensifies over control of this intelligence, so too does the demand for the data that powers it. This article explores that very topic.
We discuss the scale and urgency of the data required by AI companies and the challenges they face in acquiring it. We examine how this insatiable demand threatens both the internet as we know it and its billions of contributors. Finally, we introduce some startups leveraging cryptography to address these issues and concerns.
One caveat before diving in: this article is written from the perspective of training large language models (LLMs), not all AI systems. Therefore, I often use "AI" and "LLMs" interchangeably.
Data on Display
LLMs require three main resources: computing power, energy, and data. Backed by massive capital, corporations, governments, and startups are simultaneously competing for these resources. Among the three, the race for computing power has been the most visible—partly due to the rapid rise in NVIDIA's stock price.
Training LLMs requires vast numbers of specialized graphics processing units (GPUs), particularly NVIDIA’s A100, H100, and the upcoming B100 models. These aren’t devices you can buy off Amazon or at your local computer store—they cost tens of thousands of dollars each. NVIDIA controls how these scarce resources are allocated among AI labs, startups, data centers, and hyperscale customers.
In the 18 months following ChatGPT’s launch, GPU demand far exceeded supply, with wait times reaching up to 11 months. However, as some startups shut down, training algorithms and model architectures improve, other companies release specialized chips, and NVIDIA scales production, supply-demand dynamics are normalizing and prices are falling.
Next comes energy. Running GPUs in data centers consumes enormous amounts of power. According to some estimates, data centers could consume 4.5% of global electricity by 2030. As this surge strains existing grids, tech companies are exploring alternative energy solutions. Amazon recently acquired a nuclear-powered data center campus for $650 million. Microsoft has hired a head of nuclear technology. OpenAI’s Sam Altman supports energy startups like Helion, Exowatt, and Oklo.
From the standpoint of training AI models, both energy and computing power are commodities. Choosing a B100 over an H100, or nuclear over traditional energy, may make training cheaper, faster, or more efficient—but it won’t affect the model’s quality. In other words, in the race to build the smartest, most human-like AI models, energy and compute are table stakes, not differentiators.
The key resource is data.
James Betker, a research engineer at OpenAI, claims to have trained more generative models than anyone else has the right to. In a blog post, he stated, “Given enough weights and training time, almost every model will converge to the same point when trained long enough on the same dataset.” This means what distinguishes one AI model from another is the dataset—not anything else.
When we refer to a model as “ChatGPT,” “Claude,” “Mistral,” or “Lambda,” we’re not talking about its architecture, the GPUs used, or the energy consumed—we’re referring to the dataset it was trained on.
If data is food for AI training, then models are what they eat.
How much data does it take to train a state-of-the-art generative model? The answer is—a lot.
GPT-4, still considered the best large language model over a year after its release, is estimated to have been trained on 12 trillion tokens (or roughly 9 trillion words). This data came from crawling publicly available internet sources, including Wikipedia, Reddit, Common Crawl (a free, open repository of web crawl data), over a million hours of transcribed YouTube videos, and code platforms like GitHub and Stack Overflow.
If that sounds like a lot, hold on. In generative AI, there’s something called the “Chinchilla Scaling Laws,” which suggests that for a given compute budget, training a smaller model on a larger dataset is more effective than training a larger model on a smaller dataset. If we extrapolate the compute budgets AI companies plan to use for next-generation models (like GPT-5 and Llama-4), we find these models will require five to six times more compute and be trained on up to 100 trillion tokens.
With most publicly available internet data already scraped, indexed, and used to train existing models, where will additional data come from? This has become a frontier research problem for AI companies. There are two solutions. One is generating synthetic data—data produced directly by LLMs rather than humans. However, the usefulness of such data in making models smarter remains unproven.
The other approach is simply finding high-quality data instead of synthesizing it. Yet obtaining additional data is challenging, especially as AI companies face problems that threaten not only future model training but also the effectiveness of current models.
The first data issue involves legal concerns. Although AI companies claim to use “publicly available data,” much of it is copyrighted. For example, the Common Crawl dataset includes millions of articles from publications like The New York Times and The Associated Press, along with other copyrighted material.
Some publishers and creators are taking legal action against AI companies alleging copyright and intellectual property violations. The New York Times sued OpenAI and Microsoft for “illegally copying and using The Times’ unique valuable works.” A group of programmers filed a class-action lawsuit challenging the legality of training GitHub Copilot (a popular AI coding assistant) on open-source code.
Comedian Sarah Silverman and author Paul Tremblay have also sued AI companies for using their works without permission.
Others are embracing change through collaboration with AI firms. The Associated Press, the Financial Times, and Axel Springer have all signed content licensing agreements with OpenAI. Apple is exploring similar deals with news organizations like Condé Nast and NBC. Google agreed to pay Reddit $60 million annually for API access for model training, while Stack Overflow reached a similar agreement with OpenAI. Meta reportedly considered buying publisher Simon & Schuster outright.
These arrangements coincide with the second challenge AI companies face—the closing of the open web.
Internet forums and social media sites have realized the value their platform data brings when used by AI companies to train models. Before striking deals with Google (and potentially others), Reddit began charging for its previously free API, killing off its popular third-party clients. Similarly, Twitter restricted API access and raised prices, with Elon Musk using Twitter data to train models for his own AI company, xAI.
Even smaller publications, fan fiction forums, and niche corners of the internet that produce content freely available to all (monetized, if at all, via ads) are now shutting down. The internet was once envisioned as a magical cyberspace where everyone could find tribes sharing their unique interests and quirks. That magic seems to be slowly fading away.
This convergence of litigation threats, multi-million-dollar content deals, and the closure of the open web has two implications:
-
First, the data wars heavily favor Big Tech. Startups and small companies lack access to previously available APIs and cannot afford the multimillion-dollar fees needed to license data without legal risk. This clearly has centralizing effects—those with money can buy the best data, build the best models, and grow even wealthier.
-
Second, the business models of user-generated content platforms are increasingly misaligned with users. Platforms like Reddit and Stack Overflow rely on contributions from millions of unpaid human creators and moderators. Yet when these platforms sign multimillion-dollar deals with AI companies, they neither compensate nor seek consent from users—without whom there would be no data to sell.
Both Reddit and Stack Overflow experienced significant user strikes over these decisions. The Federal Trade Commission (FTC) has also launched an investigation into Reddit for selling, licensing, and sharing user posts with external organizations for AI model training.
These issues raise relevant questions about the future of training next-generation AI models and the fate of internet content. Under current conditions, the outlook doesn’t look optimistic. Can cryptographic solutions offer a fairer playing field for smaller companies and internet users, addressing some of these problems?
Data Pipelines
Training AI models and building useful applications are complex and expensive endeavors requiring months of planning, resource allocation, and execution. These processes involve multiple stages, each with distinct goals and data requirements.
Let’s break them down to understand how cryptographic technologies fit into the broader AI puzzle.
Pretraining
Pretraining is the first and most resource-intensive step in LLM training, forming the foundation of the model. During this phase, AI models are trained on vast amounts of unlabeled text to capture general knowledge about the world and patterns of language usage. When we say GPT-4 was trained on 12 trillion tokens, we mean the data used during pretraining.
To understand why pretraining forms the bedrock of LLMs, we need a high-level overview of how they work. Note that this is a simplified explanation—you can find deeper dives in Jon Stokes’ excellent piece, Andrej Karpathy’s engaging video, or Stephen Wolfram’s outstanding book.
LLMs use a statistical technique called Next-Token Prediction. Simply put, given a sequence of tokens (i.e., words), the model tries to predict the next most likely token. This process repeats to form complete responses. Thus, you can think of large language models as “completion machines.”
Let’s illustrate this with an example.
When I ask ChatGPT “What direction does the sun rise from?” it first predicts the word “the,” then sequentially predicts each word in “the sun rises from the East.” But where do these predictions come from? How does ChatGPT determine that “the East” should follow “the sun rises from” rather than “the West,” “the North,” or “Amsterdam”? In other words, how does it know “the East” is statistically more likely than other options?
The answer lies in learning statistical patterns from vast amounts of high-quality training data. Across all text on the internet, which phrase is more likely to appear—"the sun rises from the east" or "the sun rises from the west"? The latter might appear in specific contexts—literary metaphors (“It’s as absurd as believing the sun rises from the west”) or discussions about other planets (like Venus, where the sun does rise from the west). But overall, the former is far more common.
Through repeated next-token prediction, LLMs develop a general worldview (what we call common sense) and an understanding of linguistic rules and patterns. Another way to view LLMs is as compressed versions of the internet. This also helps explain why data must be both massive (more patterns to learn from) and high-quality (greater accuracy in pattern recognition).
But as discussed earlier, AI companies are running out of data to train ever-larger models. The rate at which training data demand grows far exceeds the pace at which new data is generated on the open web. With lawsuits looming and major forums closing, AI companies face serious challenges.
For smaller companies, the problem is worse—they can’t afford multimillion-dollar deals with proprietary data providers like Reddit.
Enter Grass, a decentralized residential proxy provider aiming to solve these data problems. They describe themselves as the “data layer for AI.” Let’s first understand what a residential proxy provider is.
The internet is the best source of training data, and web scraping is the preferred method companies use to acquire it. In practice, scraping software runs from data centers for scalability, convenience, and efficiency. But companies holding valuable data don’t want it used to train AI models (unless they’re paid). To enforce restrictions, they typically block IP addresses associated with known data centers, preventing large-scale scraping.
That’s where residential proxy providers come in. Websites usually only block IPs from known data centers, not regular internet users like you and me—making our personal internet connections, or residential IPs, valuable. Residential proxy providers aggregate millions of such connections to scrape data at scale for AI companies.
However, centralized residential proxy providers operate opaquely. They rarely disclose their intentions. Users might hesitate to share bandwidth if they knew how it was being used. Worse, they might demand compensation for bandwidth consumed, cutting into profits.
To protect their margins, residential proxy providers bundle bandwidth-consuming code into widely distributed free apps—mobile utilities (like calculators and voice recorders), VPN providers, or even consumer TV screensavers. Users think they’re getting a free product, while a third-party proxy provider silently consumes their bandwidth (details buried deep in rarely-read terms of service).
Eventually, some of this data flows to AI companies, who use it to train models and create value for themselves.
Andrej Radonjic, while running his own residential proxy business, recognized the unethical nature of these practices and their unfairness to users. Seeing advances in cryptography, he identified a path toward a fairer solution. That’s how Grass was founded in late 2022. Weeks later, ChatGPT launched, changing the world—and placing Grass at exactly the right place and time.
Unlike the covert tactics used by other residential proxy providers, Grass explicitly informs users that their bandwidth will be used to train AI models. In return, users are directly rewarded. This flips the traditional model: by voluntarily contributing bandwidth and becoming partial owners of the network, users shift from passive participants to active stakeholders—increasing network reliability and benefiting from AI-generated value.
Grass’s growth has been remarkable. Since launching in June 2023, they’ve amassed over 2 million active users who run nodes and contribute bandwidth via browser extensions or mobile apps. This growth occurred organically, without marketing spend, fueled by a highly successful referral program.
Using Grass’s services allows companies—including major AI labs and open-source startups—to access scraped training data at lower costs. Meanwhile, everyday users earn rewards for sharing their internet connection and become part of the growing AI economy.
Beyond raw scraped data, Grass offers clients additional services.
First, they transform unstructured web pages into structured data that AI models can easily process. This step—called data cleaning—is typically a resource-intensive task handled internally by AI labs. By delivering clean, structured datasets, Grass enhances its value to customers. Additionally, Grass trains an open-source LLM to automate the scraping, preparation, and labeling of data.
Second, Grass bundles datasets with undeniable provenance proofs. Given the importance of high-quality data to AI models, ensuring datasets haven’t been tampered with by malicious websites or proxy providers is critical.
The severity of this issue is reflected in initiatives like the Data & Trust Alliance, a nonprofit coalition of over 20 companies including Meta, IBM, and Walmart working together to establish data provenance standards to help organizations assess whether a dataset is suitable and trustworthy.
Grass is doing something similar. Each time a Grass node scrapes a webpage, it records metadata verifying the origin of that page. These provenance proofs are stored on-chain and shared with clients (who can further share them with their users).
Although Grass is building on Solana—one of the highest-throughput blockchains—storing provenance data for every scrape on L1 is impractical. So Grass is building a rollup (one of the first on Solana) that uses ZK processors to batch provenance proofs before publishing them to Solana. This rollup, which Grass calls the “data layer for AI,” serves as a ledger for all scraped data.
Grass’s Web3-first approach gives it several advantages over centralized competitors. First, by rewarding users directly for sharing bandwidth, they distribute AI-generated value more fairly (while saving costs on paying app developers to bundle their code). Second, they can charge a premium for providing clients “legitimate traffic”—a highly valued commodity in the industry.
Another protocol working in the space of “legitimate traffic” is Masa. The network allows users to share login credentials from social media platforms like Reddit, Twitter, or TikTok. Nodes then scrape highly contextual, real-time updates from these platforms. The advantage here is that the collected data mirrors what ordinary users see on their feeds. In real time, you gain rich datasets explaining sentiment or content going viral.
These datasets serve two primary purposes:
-
Finance – If you can observe what thousands of people are seeing on their social feeds, you can develop trading strategies based on that data. Autonomous agents trained on Masa’s datasets can leverage sentiment analysis.
-
Social – The rise of AI companions (or tools like Replika) means we need datasets that mimic human conversation. These conversations also need to stay current. Masa’s data streams can train agents capable of meaningfully discussing trending Twitter topics.
Masa’s approach enables access to information from closed gardens (like Twitter) through user consent, making it available to developers building applications. This socially oriented data collection also enables region-specific language datasets.
For example, a Hindi-speaking bot could be trained on data pulled from social networks operating in Hindi. The application possibilities opened by such networks remain largely unexplored.
Model Alignment
A pretrained LLM is far from ready for production use. Consider this: the model currently only knows how to predict the next word in a sequence—and nothing else. If you give a pretrained model a prompt like “Who is Satoshi Nakamoto,” any of these could be valid responses:
-
Complete the question: Satoshi Nakamoto?
-
Turn the phrase into a sentence: is a mystery that has puzzled Bitcoin believers for years.
-
Actually answer the question: Satoshi Nakamoto is the pseudonymous person or group who created Bitcoin, the first decentralized cryptocurrency, and its underlying blockchain technology.
An LLM designed to provide helpful answers would deliver the third response. But a pretrained model doesn’t consistently or correctly produce such outputs. In fact, it often randomly spews meaningless text to end users. At worst, it generates false, toxic, or harmful information under the guise of confidentiality. When this happens, the model is said to “hallucinate.”
This is how a pretrained GPT-3 answered the question
The goal of model alignment is to make pretrained models useful to end users. In other words, to transform them from mere statistical text completion tools into conversational agents that understand and align with user intent, producing coherent and helpful dialogue.
Conversational Fine-Tuning
The first step in this process is conversational fine-tuning. Fine-tuning refers to further training a pretrained machine learning model on a smaller, targeted dataset to adapt it to a specific task or use case. For LLMs, this specific use case is conducting human-like conversations. Naturally, this fine-tuning dataset consists of human-generated prompt-response pairs demonstrating desired behavior.
These datasets cover various conversation types (Q&A, summarization, translation, code generation) and are typically created by highly educated humans (sometimes called AI tutors) with strong language skills and domain expertise.
State-of-the-art models like GPT-4 are estimated to have been trained on ~100,000 such prompt-response pairs.
Example of a prompt-response pair
Reinforcement Learning from Human Feedback (RLHF)
Think of this step as akin to training a pet dog: reward good behavior, punish bad behavior. The model receives a prompt, and its response is shared with human annotators who score the output based on accuracy and quality (e.g., 1–5 scale). An alternative version of RLHF generates a prompt and multiple responses, asking human annotators to rank them from best to worst.
Example of an RLHF task
RLHF aims to guide the model toward behaviors aligned with human preferences and expectations. In fact, if you’re a ChatGPT user, OpenAI uses you as an RLHF annotator! This happens when the model sometimes generates two responses and asks you to pick the better one.
Even simple thumbs-up or thumbs-down icons prompting you to rate a response’s usefulness serve as RLHF training for the model.
We rarely consider the millions of hours of human labor behind AI models when using them. And it’s not just LLMs that require this. Historically, even traditional machine learning use cases—from content moderation to autonomous driving and tumor detection—have relied heavily on human involvement in data annotation. (This excellent 2019 New York Times story reveals the behind-the-scenes operations of iAgent’s Indian office, a firm specializing in human annotation.)
Fei-Fei Li’s use of Mechanical Turk to build the ImageNet database was dubbed “artificial artificial intelligence” by Jeff Bezos because of the invisible role its workers played in AI training.
In a bizarre story earlier this year, Amazon’s Just Walk Out stores—where customers
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
@Decentralisedco
