In-Depth Research: Is It Feasible to Crowdfund an AI Model Using Cryptographic Incentives?
TechFlow Selected TechFlow Selected
In-Depth Research: Is It Feasible to Crowdfund an AI Model Using Cryptographic Incentives?
This report examines the current state of large model training and related costs.
Navigating Web3 tides with focused insightsAuthor: Jeff Amico
Translation: TechFlow
Introduction
During the pandemic, Folding@home achieved a major milestone. The research project reached 2.4 exaFLOPS of computing power, contributed by over 2 million volunteer devices worldwide. This represented fifteen times the processing capacity of the world’s most powerful supercomputer at that time, enabling scientists to massively simulate the dynamics of COVID proteins. Their work advanced our understanding of the virus and its pathogenic mechanisms, especially during the early stages of the pandemic.
Global distribution of Folding@home users, 2021
Folding@home builds on a long history of volunteer computing—crowdsourcing computational resources to solve large-scale problems. This idea gained widespread attention with SETI@home in the 1990s, which pooled over 5 million volunteer computers to search for extraterrestrial life. Since then, the concept has been applied across fields including astrophysics, molecular biology, mathematics, cryptography, and gaming. In each case, collective effort amplifies individual capabilities far beyond what any single entity could achieve alone, accelerating progress through more open and collaborative research.
Many have wondered whether this crowdsourced model can be applied to deep learning. In other words, can we train a large neural network among the general public? Training frontier models is one of the most computationally intensive tasks in human history. Like many @home projects, the current cost exceeds what only the largest players can afford. This risks stifling future progress as we rely on fewer and fewer companies to deliver breakthroughs. It also centralizes control over AI systems in the hands of a few. Regardless of your stance on the technology, this is a concerning trajectory.
Most critics dismiss decentralized training as incompatible with current techniques. However, this view is increasingly outdated. New technologies have emerged that reduce inter-node communication needs, enabling efficient training even on devices with poor network connectivity. These include DiLoCo, SWARM Parallelism, lo-fi, and decentralized training of foundation models in heterogeneous environments. Many are fault-tolerant and support heterogeneous computing. Some new architectures are specifically designed for decentralized networks, such as DiPaCo and decentralized mixture-of-experts models.
We’re also seeing cryptographic primitives mature, enabling global coordination of resources. These support applications like digital currencies, cross-border payments, and prediction markets. Unlike earlier volunteer efforts, these networks can aggregate staggering amounts of compute—often orders of magnitude larger than the largest cloud training clusters currently envisioned.
Together, these elements form a new paradigm for model training. This paradigm taps into global computational resources—including vast numbers of edge devices that could be harnessed if connected. It will lower costs for most training workloads by introducing competitive dynamics. It may unlock new forms of training, making model development collaborative and modular rather than isolated and monolithic. Models could learn in real time using computation and data from the crowd. Individuals could own parts of the models they help create. Researchers could once again openly share novel findings without needing to monetize discoveries to offset expensive compute budgets.
This report examines the current state and associated costs of frontier model training. It reviews past distributed computing efforts—from SETI to Folding to BOINC—to inspire alternative paths forward. It discusses historical challenges of decentralized training and turns to recent breakthroughs that may help overcome them. Finally, it outlines future opportunities and challenges.
The State of Frontier Model Training
The cost of frontier model training has become unaffordable for all but the largest players. This trend isn’t new, but according to actual trends, it's worsening as leading labs continue to test the limits of scaling assumptions. Reports suggest OpenAI spent over $3 billion on training this year. Anthropic predicts we’ll begin $10 billion training runs by 2025, with $100 billion models not far behind.
This trend drives industry consolidation, as only a handful of companies can afford to participate. This raises critical policy questions: Can we accept a future where all leading AI systems are controlled by just one or two firms? It also slows progress—a reality evident in the research community, where smaller labs lack access to the compute needed for scaling experiments. Industry leaders themselves have highlighted this issue:
Meta’s Joe Spisak: To truly understand the capabilities of [models] architecturally, you have to explore at scale, and I think that's what's missing in the ecosystem today. If you look at academia—there are brilliant people in academia—but they don't have access to compute resources, and that becomes a problem because they have these great ideas but no way to actually realize them at the level required.
Together’s Max Ryabinin: The need for expensive hardware puts immense pressure on the research community. Most researchers cannot participate in large neural network development because running necessary experiments is too costly for them. If we keep increasing model size by scaling up, eventually only a few entities will be able to compete.
Google’s Francois Chollet: We know LLMs haven't achieved AGI. Meanwhile, progress toward AGI has stalled. The limitations we face with LLMs are exactly the same as those we faced five years ago. We need new ideas and breakthroughs. I believe the next breakthrough will likely come from external teams, while all big labs are busy training ever-larger LLMs.
Some remain skeptical, arguing that hardware improvements and cloud capital expenditures will resolve the issue. But this seems unrealistic. By the end of this decade, next-generation Nvidia chips may offer up to ten times more FLOPs than today’s H100s, reducing per-FLOP prices by 80–90%. Similarly, total FLOP supply is expected to increase roughly twentyfold, along with improved networking and infrastructure. All of this will improve training efficiency per dollar.
Source: SemiAnalysis AI Cloud TCO Model
At the same time, total FLOP demand will rise sharply as labs aim to scale further. If the decade-long trend in training compute continues, frontier training FLOPs could reach ~2e29 by 2030. Training at this scale would require approximately 20 million H100-equivalent GPUs, based on current runtimes and utilization. Assuming multiple frontier labs, total required FLOPS would be several times higher due to shared supply. EpochAI estimates we’ll need around 100 million H100-equivalent GPUs by then—about 50 times the 2024 shipment volume. SemiAnalysis makes a similar prediction, suggesting frontier training demand and GPU supply grow roughly in tandem during this period.
Supply conditions could tighten for various reasons. Manufacturing bottlenecks could delay shipments—a common occurrence. We might fail to generate enough energy to power data centers. Connecting energy sources to the grid could prove difficult. Increasing scrutiny of capital spending could ultimately lead the industry to downsize. At best, our current approach allows only a few companies to drive progress—and that may not be sufficient.
Clearly, we need a new approach—one that doesn’t require endless expansion of data centers, capital expenditure, and energy consumption to find the next breakthrough. Instead, it should efficiently utilize existing infrastructure and scale flexibly with demand fluctuations. This would enable more experimentation in research, as training runs wouldn’t need to justify billion-dollar compute budgets. Once freed from this constraint, we could move beyond the current LLM paradigm—which many believe is insufficient for achieving artificial general intelligence (AGI). To understand what an alternative might look like, we can draw inspiration from past distributed computing practices.
Collective Computing: A Brief History
SETI@home popularized the concept in 1999, allowing millions of participants to analyze radio signals for signs of extraterrestrial intelligence. SETI collected electromagnetic data from the Arecibo telescope, split it into batches, and sent them via the internet to users. Participants analyzed the data during idle time and returned results. No communication between users was needed; batches could be processed independently, enabling highly parallel processing. At its peak, SETI@home had over 5 million participants and surpassed the processing power of the world’s fastest supercomputers. It officially shut down in March 2020, but its success inspired subsequent volunteer computing movements.
Folding@home carried the torch in 2000, using edge computing to simulate protein folding related to diseases like Alzheimer’s, cancer, and Parkinson’s. Volunteers ran simulations on personal computers during idle time, helping researchers study how misfolded proteins cause illness. At various points in its history, its computational power exceeded that of the world’s largest supercomputers—especially in the late 2000s and during the pandemic, when it became the first distributed computing project to surpass one exaFLOP. Since inception, Folding researchers have published over 200 peer-reviewed papers, each relying on volunteer-provided computing power.
The Berkeley Open Infrastructure for Network Computing (BOINC) popularized the concept in 2002, providing a platform for crowdsourced computing across diverse research initiatives. It supports projects like SETI@home and Folding@home, as well as new efforts in astrophysics, molecular biology, mathematics, and cryptography. As of 2024, BOINC lists 30 active projects and nearly 1,000 scientific publications leveraging its network.
Beyond science, volunteer computing has trained game engines for Go (LeelaZero, KataGo) and chess (Stockfish, LeelaChessZero). LeelaZero trained via volunteer computing from 2017 to 2021, playing over ten million games against itself to become one of today’s strongest Go engines. Similarly, Stockfish has continuously trained on volunteer networks since 2013, making it one of the most popular and powerful chess engines.
Challenges for Deep Learning
But can we apply this model to deep learning? Can we network edge devices globally to create a low-cost public training cluster? Consumer hardware—from Apple laptops to Nvidia gaming GPUs—is becoming increasingly capable for deep learning. In many cases, their performance per dollar even exceeds that of data center GPUs.
Yet, effectively utilizing these resources in a distributed setting requires overcoming significant challenges.
First, current distributed training techniques assume frequent communication between nodes.
State-of-the-art models are now so large that training must be split across thousands of GPUs. This is achieved through various parallelization techniques, typically partitioning the model, dataset, or both across available GPUs. This usually demands high-bandwidth, low-latency networks; otherwise, nodes sit idle waiting for data.
For example, Distributed Data Parallel (DDP) distributes the dataset across GPUs. Each GPU trains the full model on its portion and shares gradient updates to generate new model weights. This incurs relatively low communication overhead, as nodes share gradients only after each backward pass, and communication can overlap partially with computation. However, DDP works only for smaller models, requiring each GPU to store the entire model’s weights, activations, and optimizer states in memory. For instance, GPT-4 required over 10TB of memory during training, while a single H100 has only 80GB.
To address this, models are split across GPUs using techniques like Tensor Parallelism, which partitions weights within layers, allowing each GPU to perform operations and pass outputs to others. This reduces per-GPU memory needs but requires constant communication, demanding high-bandwidth, low-latency connections for efficiency.
Pipeline Parallelism assigns layers of the model to different GPUs, each performing its task and passing updates to the next. While requiring less communication than tensor parallelism, it can suffer from "bubbles"—idle periods where downstream GPUs wait for upstream ones to finish.
To tackle these issues, methods like ZeRO (Zero Redundancy Optimizer) were developed—memory optimization techniques that trade increased communication for reduced memory usage, enabling larger models on specific devices. ZeRO reduces memory demands by partitioning model parameters, gradients, and optimizer states across GPUs, but relies heavily on communication to retrieve partitioned data. It underpins popular approaches like Fully Sharded Data Parallel (FSDP) and DeepSpeed.
These techniques are often combined in large-scale training—a practice known as 3D Parallelism. In such setups, tensor parallelism handles intra-server weight distribution due to high communication needs within split layers. Pipeline parallelism spans servers (within the same data center island), requiring less communication. Data parallelism or FSDP splits datasets across server islands, tolerating longer latencies via asynchronous updates or gradient compression. Meta used this combination to train Llama 3.1, illustrated below.
These methods pose core challenges for decentralized training networks relying on slower, more variable consumer internet connections. In such environments, communication costs quickly outweigh the benefits of edge compute, as devices often sit idle waiting for data. For instance, training a half-precision model with 1 billion parameters using DDP requires each GPU to share 2GB of data per step. With typical internet bandwidth (~1 Gbps) and no overlap between computation and communication, transmitting gradients takes at least 16 seconds—leading to significant idle time. Techniques like tensor parallelism, requiring even more communication, fare worse.
Second, current training lacks fault tolerance. Like any distributed system, larger clusters are more prone to failures. But this is particularly severe in training, where current methods are largely synchronous—requiring GPUs to coordinate closely. A single GPU failure among thousands can halt the entire process, forcing others to restart. Sometimes GPUs don’t fail outright but slow down, dragging down thousands of others. Given the scale of modern clusters, this can mean tens or hundreds of millions in additional costs.
Meta detailed these issues during Llama training, experiencing over 400 unplanned interruptions—about 8 per day—mostly due to GPU or host hardware failures. This led to GPU utilization rates of only 38–43%. OpenAI fared worse during GPT-4 training, with utilization at 32–36%, also due to frequent failures.
In other words, even top labs operating in fully optimized environments—with homogeneous, cutting-edge hardware, networking, power, and cooling—struggle to exceed 40% utilization. This stems mainly from hardware and network issues, which would be far worse in edge training environments with uneven performance, bandwidth, latency, and reliability. Not to mention, decentralized networks are vulnerable to malicious actors seeking to sabotage projects or cheat on specific tasks. Even purely volunteer networks like SETI@home have seen cheating among participants.
Third, frontier model training demands massive computational power. While projects like SETI and Folding achieved impressive scale, they pale compared to today’s training requirements. GPT-4 was trained on a cluster of 20,000 A100s, peaking at 6.28 exaFLOPS in half-precision—three times Folding@home’s peak. Llama 405B trained on 16,000 H100s, reaching 15.8 exaFLOPS—seven times Folding’s peak. As multiple labs plan clusters exceeding 100,000 H100s, each capable of up to 99 exaFLOPS, this gap will widen further.
This makes sense—@home projects are volunteer-driven. Contributors donate memory and processor cycles, bearing the associated costs. This naturally limits their scale relative to commercial efforts.
Recent Advances
While these issues historically hindered decentralized training, they now appear surmountable. New techniques reduce inter-node communication needs, enabling efficient training over internet-connected devices. Many originate from large labs aiming to scale training across data centers, necessitating efficient cross-data-center communication. We’ve also seen advances in fault-tolerant training and cryptographic incentive systems, supporting large-scale training in edge environments.
Efficient Communication Techniques
DiLoCo, a recent Google research project, reduces communication overhead by performing local optimization before sharing updated model states. Their method—based on earlier federated learning work—achieves performance comparable to traditional synchronous training while reducing node-to-node communication by 500x. Since then, the approach has been replicated and extended to train larger models (>1B parameters), including asynchronous variants that allow nodes to share updates at different times—better adapting to edge hardware with varying speeds.
Other data-parallel methods like lo-fi and DisTrO further reduce communication costs. Lo-fi proposes fully local fine-tuning—nodes train independently and share weights only at the end. This matches baseline performance when fine-tuning language models >1B parameters, eliminating communication entirely. A preliminary report claims DisTrO uses a novel distributed optimizer that reduces communication needs by four to five orders of magnitude, though this remains unverified.
New model-parallel approaches also enable greater scalability. DiPaCo (also from Google) divides models into modules containing different expert subnetworks for task-specific training. Training data is sharded into “paths”—sequences of experts corresponding to each sample. Given a shard, each worker can almost independently train its path, except for shared modules handled via DiLoCo. This architecture cuts training time for billion-parameter models by over half.
SWARM Parallelism and Decentralized Training of Foundation Models in Heterogeneous Environments (DTFMHE) propose model-parallel methods for large models in heterogeneous settings. SWARM finds that as models grow, pipeline parallelism communication constraints diminish, enabling effective training at lower bandwidth and higher latency. To apply this in heterogeneous environments, they use temporary “pipeline links” between nodes, dynamically updated each iteration. Outputs can be routed to any peer in the next stage, allowing dynamic rerouting if some peers are faster or disconnect—ensuring training continues as long as each stage has at least one active participant. They demonstrated this by training a >1B parameter model on low-cost, heterogeneous GPUs with slow interconnects (shown below).
DTFMHE similarly proposes a novel scheduling algorithm combining pipeline and data parallelism to train large models across devices on three continents. Despite network speeds 100x slower than standard Deepspeed, their method was only 1.7–3.5x slower than standard Deepspeed in data centers. Like SWARM, DTFMHE shows communication costs can be effectively hidden as model size increases—even in geographically distributed networks—by increasing hidden layer sizes or adding more layers per pipeline stage.
Fault Tolerance
Many of the above data-parallel methods are inherently fault-tolerant, as each node stores the full model in memory. This redundancy means nodes can continue working even if others fail—an essential feature for decentralized training where nodes are unreliable, heterogeneous, or potentially malicious. However, pure data parallelism only suits small models, limiting size to the smallest node’s memory capacity.
To address this, fault-tolerant techniques for model-parallel (or hybrid) training have emerged. SWARM mitigates peer failures by prioritizing stable, low-latency peers and dynamically rerouting pipeline tasks upon failure. Others, like Oobleck, use multiple “pipeline templates” for redundancy during partial node failures. Though tested in data centers, Oobleck’s reliability guarantees apply equally well to decentralized environments.
New architectures like the Decentralized Mixture of Experts (DMoE) also support fault-tolerant training in decentralized settings. Similar to traditional mixture-of-experts models, DMoE consists of independent “expert” networks distributed across worker nodes. DMoE uses a distributed hash table to track and aggregate asynchronous updates in a decentralized manner. This mechanism—also used in SWARM—is resilient to node failures, as it can exclude non-responsive experts from averaging calculations.
Scaling Up
Finally, cryptographic incentive systems like those used by Bitcoin and Ethereum can help achieve the necessary scale. These networks crowdsource compute by rewarding contributors with native assets that appreciate with adoption. This design incentivizes early participation with generous rewards, gradually tapering off once the network reaches minimal viability.
Certainly, pitfalls exist—primarily over-incentivizing supply without matching demand. If the underlying network isn’t sufficiently decentralized, regulatory issues may arise. Yet, when well-designed, decentralized incentive systems can sustain significant scale over time.
For example, Bitcoin consumes about 150 terawatt-hours (TWh) annually—two orders of magnitude higher than the projected power use of the largest AI training clusters (100,000 H100s running full-time for a year). For reference, OpenAI trained GPT-4 on 20,000 A100s, Meta’s flagship Llama 405B on 16,000 H100s. At its peak, Ethereum consumed ~70 TWh, spread across millions of GPUs. Even accounting for rapid growth in AI data centers, such incentivized compute networks will repeatedly surpass them in scale.
Of course, not all computation is interchangeable—training has unique requirements to consider. Still, these networks demonstrate the scale achievable through such mechanisms.
The Path Forward
Bringing these pieces together, we see the beginnings of a new path forward.
Soon, new training techniques will allow us to transcend data center boundaries, as devices no longer need to be co-located to function. This will take time—current decentralized training methods still operate at smaller scales, mostly within 1–2 billion parameters, far below models like GPT-4. Further breakthroughs are needed to scale these methods without sacrificing key properties like communication efficiency and fault tolerance. Alternatively, we may need new model architectures fundamentally different from today’s large monoliths—possibly smaller, more modular, designed to run on edge devices rather than in the cloud.
Regardless, continued progress in this direction is highly plausible. The unsustainable cost of our current approach creates strong market incentives for innovation. We’re already seeing this trend—manufacturers like Apple are building more powerful edge devices to run workloads locally instead of relying on the cloud. Support for open-source solutions is growing—even within companies like Meta—to foster more decentralized R&D. These trends will only accelerate over time.
Meanwhile, we also need new network infrastructure to connect edge devices for such use. These include laptops, gaming desktops, and eventually even phones with high-performance GPUs and large memory. This would let us build a “global cluster”—low-cost, always-on computing power capable of parallelizing training tasks. This too is challenging and requires advances across multiple domains.
We need better scheduling techniques for heterogeneous environments. No method yet exists to automatically parallelize models optimally, especially when devices can join or leave anytime. This is a crucial next step to optimize training while preserving the scale advantages of edge networks.
We must also address the general complexity of decentralized networks. To maximize scale, networks should be built as open protocols—standardized rules governing participant interactions, much like TCP/IP but for machine learning computation. This would allow any compliant device to join, regardless of owner or location. It ensures neutrality, letting users train whatever models they prefer.
While maximizing scale, this also requires mechanisms to verify training correctness without relying on a single authority. This is critical due to inherent incentives to cheat—e.g., claiming completed work to earn rewards without doing so. Because different devices execute ML operations differently, standard replication techniques struggle to verify correctness, making this especially hard. Solving it properly requires deep research in cryptography and related fields.
Luckily, we continue to see progress across all these fronts. Compared to just a few years ago, these challenges no longer seem insurmountable. Relative to the opportunities, they now appear quite small. Google best summarized this in their DiPaCo paper, highlighting the negative feedback loop decentralized training could break:
Advances in distributed training of machine learning models could promote simpler infrastructure construction, ultimately leading to broader availability of computational resources. Currently, infrastructure is designed around the standard approach of training large monolithic models, while model architectures are tailored to exploit current infrastructure and training methods. This feedback loop may trap the community in a misleading local minimum where computational constraints exceed actual necessity.
Perhaps most exciting is the growing enthusiasm in the research community to solve these problems. Our team at Gensyn is building the network infrastructure described above. Teams like Hivemind and BigScience are applying many of these techniques in practice. Projects like Petals, sahajBERT, and Bloom demonstrate the potential of these technologies and growing interest in community-driven machine learning. Many others are advancing research toward a more open, collaborative model training ecosystem. If you're interested in this work, please reach out to get involved.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
@gensynai
