Turing Award winners worry about becoming the "Oppenheimer" of AI
TechFlow Selected TechFlow Selected
Turing Award winners worry about becoming the "Oppenheimer" of AI
Once a founder of AI, now a pioneer of "anti-AI."
Navigating Web3 tides with focused insightsAuthor: Moonshot
In 1947, Alan Turing mentioned in a lecture: "What we want is a machine that can learn from experience."
Seventy-eight years later, the Turing Award—named after Turing and widely regarded as the Nobel Prize of computing—was awarded to two scientists who have spent their lives addressing this very challenge.
Andrew Barto and Richard Sutton jointly received the 2024 Turing Award. Nine years apart in age, they are mentor and protégé, the technical pioneers behind AlphaGo and ChatGPT, and trailblazers in the field of machine learning.
Turing Award recipients Andrew Barto and Richard Sutton
Image source: ACM A.M. Turing Award website
Jeff Dean, Google’s chief scientist, wrote in the award citation: “The reinforcement learning techniques pioneered by Barto and Sutton directly answer Turing’s question. Their work has been key to AI advances over the past several decades. The tools they developed remain central pillars of today’s AI boom… Google is proud to sponsor the ACM A.M. Turing Award.”
Google is the sole sponsor of the Turing Award's $1 million prize.
Yet after receiving the honor, these two scientists stepped into the spotlight not to celebrate, but to criticize major AI companies. In interviews with the media, they delivered an unconventional acceptance speech: today’s AI firms operate under “commercial incentives” rather than a commitment to genuine research, “erecting an untested bridge and inviting people to cross it for testing.”
Notably, the last time the Turing Award was given to researchers in artificial intelligence was in 2018, when Yoshua Bengio, Geoffrey Hinton, and Yann LeCun were honored for their contributions to deep learning.
2018 Turing Award recipients
Image source: eurekalert
Among them, Yoshua Bengio and Geoffrey Hinton (who also won the 2024 Nobel Prize in Physics) have frequently warned global society and the scientific community in recent years about the potential misuse of AI by large corporations.
Hinton even resigned from Google so he could speak freely. Sutton, one of this year’s awardees, served as a research scientist at DeepMind from 2017 to 2023.
As computing’s highest honor continues to be bestowed upon foundational figures in AI, a striking pattern emerges:
Why do these towering scientists so often turn to sound the alarm on AI once they stand in the limelight?
The Bridge Builders of Artificial Intelligence
If Alan Turing was the pioneer of artificial intelligence, then Andrew Barto and Richard Sutton are the “bridge builders” along that path.
At a moment when AI is accelerating rapidly and being celebrated globally, they are re-examining whether the bridges they constructed can safely carry humanity across.
Perhaps the answer lies within their half-century-long academic journey—only by retracing how they built “machine learning” can we understand why they now warn against “technological失控.”
Image source: Carnegie Mellon University
In 1950, Alan Turing opened his seminal paper “Computing Machinery and Intelligence” with a philosophical and technical question:
“Can machines think?”
To explore this, Turing devised the “imitation game,” later known widely as the “Turing Test.”
He also proposed that machine intelligence could be acquired through learning, not merely pre-programming. He envisioned a “child machine”—a system trained through experience, gradually learning like a child.
The core goal of artificial intelligence is to build agents capable of perception and optimal action. The measure of intelligence lies in an agent’s ability to judge that “some actions are better than others.”
This is precisely the aim of machine learning: providing feedback after actions so machines can autonomously learn from experience. In other words, Turing’s conception of reward- and punishment-based machine learning resembles Pavlov’s dog training.
Getting stronger the more I lose in games is also a form of "reinforcement learning"
Image source: zequance.ai
The path toward machine learning initiated by Turing remained largely uncharted for three decades—until a mentor and student duo built the bridge: Reinforcement Learning (RL).
In 1977, Andrew Barto, inspired by psychology and neuroscience, began exploring a new theory of human intelligence: neurons act like “hedonists.” Billions of neurons in the human brain each strive to maximize pleasure (reward) and minimize pain (punishment). Rather than mechanically receiving and transmitting signals, neurons reinforce activity patterns that lead to positive feedback, collectively driving the learning process.
In the 1980s, Barto took on his doctoral student Richard Sutton, aiming to apply this theory—of trial-and-error, feedback-driven adjustment, and optimization of behavior—to artificial intelligence. Thus, reinforcement learning was born.
"Reinforcement Learning: An Introduction" has become a classic textbook, cited nearly 80,000 times
Image source: IEEE
Together, they leveraged the mathematical foundations of Markov decision processes to develop core reinforcement learning algorithms, systematically establishing its theoretical framework. They authored the textbook *Reinforcement Learning: An Introduction*, enabling tens of thousands of researchers to enter the field. They are rightly called the fathers of reinforcement learning.
Their purpose in studying reinforcement learning was to discover efficient, accurate, high-reward, and optimal machine learning methods.
The "God Move" of Reinforcement Learning
If traditional machine learning is akin to “spoon-feeding,” then reinforcement learning is “free-range” learning.
Traditional machine learning involves feeding models vast amounts of labeled data to establish fixed input-output mappings. The classic example is showing a computer many photos of cats and dogs, explicitly labeling each, until it learns to distinguish them.
Reinforcement learning, however, operates without explicit instructions. Instead, machines learn through trial and error, guided by rewards and penalties to gradually adjust behavior and optimize outcomes. It’s like a robot learning to walk—not needing constant human feedback on “this step right, that step wrong”—but simply trying, falling, adjusting, and eventually walking independently, perhaps even developing its own unique gait.
Clearly, reinforcement learning closely mirrors human intelligence: every child learns to walk through falls, grasps objects through exploration, and acquires language through babbling and listening.
The viral "back-kicking robot" is also trained using reinforcement learning
Image source: Unitree Robotics
The defining moment for reinforcement learning came in 2016 with AlphaGo’s “God Move.” During its match against Lee Sedol, AlphaGo played its 37th move—a white stone that stunned all human observers. This unexpected move reversed the game’s momentum and secured victory.
Top Go players and commentators were baffled; in human experience, the move seemed nonsensical. Lee Sedol later admitted he had never considered such a play.
AlphaGo did not arrive at this “God Move” by memorizing game records. It emerged autonomously through countless self-play sessions—trial, error, long-term planning, and strategy optimization. This is the essence of reinforcement learning.
Lee Sedol, thrown off rhythm by AlphaGo's "God Move"
Image source: AP
Reinforcement learning has even begun to influence human intelligence. After AlphaGo revealed its “God Move,” professional players started studying and adopting AI strategies. Scientists are applying reinforcement learning principles to understand how the human brain learns. One of Barto and Sutton’s achievements was creating a computational model explaining dopamine’s role in human decision-making and learning.
Reinforcement learning excels in complex, dynamic environments where finding optimal solutions is challenging—such as Go, autonomous driving, robotics control, or engaging in nuanced conversations with humans.
These are precisely the most advanced and popular frontiers in AI today. Especially in large language models (LLMs), nearly all leading systems use RLHF (Reinforcement Learning from Human Feedback)—where humans rate model outputs, and the model improves based on that feedback.
But this is exactly what worries Barto: after building the bridge, companies are letting people test its safety by walking across it.
“Deploying software to millions of users without any safeguards is not responsible,” Barto said in a post-award interview.
“Technological progress should come with measures to control and mitigate potential negative impacts, but I don’t see these AI companies doing that,” he added.
What Are the Leading AI Scientists Really Worried About?
The discourse around AI threats persists because scientists fear losing control over the future they helped create.
In their “acceptance remarks,” Barto and Sutton express no criticism of current AI technology itself, but profound dissatisfaction with AI corporations.
In interviews, both warn that today’s AI development is driven by big companies racing to release powerful yet error-prone models. These firms raise massive funding, reinvest billions, and engage in an arms race over chips and data.
Major investment banks are revaluing the AI industry
Image source: Goldman Sachs
Indeed, according to Deutsche Bank research, tech giants’ total investment in AI is now around $340 billion—surpassing Greece’s annual GDP. Industry leader OpenAI, valued at $260 billion, is preparing for a new $40 billion funding round.
In fact, many AI experts share Barto and Sutton’s concerns.
Stephen Sinofsky, former Microsoft executive, previously stated that the AI industry is trapped in a scalability dilemma—trading money for progress—an unsustainable trend contrary to historical technological evolution, where costs typically decrease over time.
On March 7, Eric Schmidt (former Google CEO), Alex Wang (founder of Scale AI), and Dan Hendrycks (director of the Center for AI Safety) co-authored a cautionary paper.
These three tech luminaries argue that the current state of frontier AI development mirrors the nuclear arms race that led to the Manhattan Project. AI companies are quietly running their own “Manhattan Projects,” doubling investments annually for nearly a decade. Without regulatory intervention, AI could become the most destabilizing technology since the atomic bomb.
"Strategic Superintelligence" and its authors
Image source: nationalsecurity.ai
Yoshua Bengio, who won the Turing Award in 2019 for deep learning, also published a lengthy blog post warning that the AI industry now offers trillions in value for capital to chase, with the power to severely disrupt the existing world order.
Many technically trained professionals believe today’s AI industry has strayed from rigorous research, reflection on intelligence, and vigilance against misuse—instead embracing a profit-driven model fueled by massive capital, hardware, and scale.
“Building huge data centers, charging users, and making them run potentially unsafe software—that’s not a motivation I support,” Barto said in a post-award interview.
The first edition of the *International Scientific Report on Advanced Artificial Intelligence Safety*, co-authored by 75 AI experts from 30 countries, states: “Approaches to managing AGI risk often assume that AI developers and policymakers can accurately assess the capabilities and potential impacts of AGI models and systems. However, scientific understanding of AGI’s internal workings, capabilities, and societal effects remains extremely limited.”
Yoshua Bengio's warning blog post
Image source: Yoshua Bengio
It’s clear that today’s “AI threat narrative” has shifted focus—from technology itself to the corporations deploying it.
Experts are asking big companies: you spend billions, stack hardware, and race on parameters—but do you truly understand your own products? This is the origin of Barto and Sutton’s “bridge” metaphor: technology belongs to humanity, but capital belongs to corporations.
Moreover, Barto and Sutton’s life’s work—reinforcement learning—is inherently closer to human cognition and features a “black box” nature. In deep reinforcement learning, AI behavior becomes increasingly complex and inscrutable.
This is precisely the scientists’ concern: they helped nurture and witness AI’s growth, yet struggle to interpret its intentions.
The Turing Award winners who pioneered deep learning and reinforcement learning aren’t necessarily afraid of AGI (Artificial General Intelligence) advancing—but of an arms race among corporations triggering an “intelligence explosion” in the AGI domain, accidentally creating ASI (Artificial Superintelligence). The distinction isn't just technical—it determines the fate of human civilization.
An ASI surpassing human intelligence would possess information capacity, decision speed, and self-evolution capabilities far beyond human comprehension. Without extremely careful design and governance, it could become the final—and most unstoppable—technological singularity in human history.
In this era of AI frenzy, these scientists may be the most qualified to pour cold water on the hype. After all, fifty years ago, when computers were still room-sized behemoths, they had already begun researching artificial intelligence. They shaped the present from the past—and thus hold the moral authority to question the future.
Will AI leaders face an Oppenheimer-like fate?
Image source: The Economist
In a February interview with The Economist, CEOs of DeepMind and Anthropic said:
They lie awake at night fearing they might become the next Oppenheimer.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
极客公园
