Douyin cracks down on AI-generated content: what's next for AI virtual streamers?
TechFlow Selected TechFlow Selected
Douyin cracks down on AI-generated content: what's next for AI virtual streamers?
Regulations on new technologies and emerging fields do not necessarily hinder development. The issuance of "bans" often also serves to promote development through compliance.
Navigating Web3 tides with focused insightsBy Xiaosa Team
On March 27, Douyin Security Center released a "Notice on Governance of Improper Use of AI-Generated Virtual Characters," aiming to further regulate content created using AI technology on its platform—particularly short videos and live-streaming hosted by AI-generated virtual characters.
Given the unusually strong language used in this notice compared to previous statements, its release has sparked concerns among industry professionals about the future prospects of AI virtual hosts on social media platforms. So, is this representative update in platform rules a blessing or a curse for the development of AIGC? And does the newly defined "red lines" represent good or bad news for the future application of AI virtual streamers?
AI-Generated Virtual Hosts: Opportunities and Risks Coexist
Compared with early "skin-suit virtual streamers" operated via real-time motion capture by human performers ("zhong zhi ren"), today's AI-driven virtual hosts—powered by AI animation generation and voice synthesis technologies—are rapidly gaining momentum unmatched by their predecessors. While these technologies still have significant room for improvement in areas such as long-form video generation and real-time interaction, they are increasingly favored by creators in the short-video and livestream e-commerce sectors. Due to relatively fixed content formats and use cases in these industries, it is now cost-effective to create AI-generated avatars tailored to specific audience preferences and automatically generate audiovisual content based on preset prompts.
Compared to traditional live-streaming, which involves rising labor costs, limited broadcasting hours, and high risks of influencer scandals ("talent collapse"), AI virtual hosts possess inherent advantages. Especially in the post-pandemic era, with growing market demand for e-commerce livestreaming and social media traffic acquisition, substantial business opportunities have emerged. Many social platforms themselves not only support but actively encourage users to adopt AIGC tools to boost content output. Take our main subject today—Douyin—as an example. Although it does not directly offer AI-generated character creation services, it has long launched integrated creative modules featuring functions like "AI image-to-image" and "AI video generation."
Clearly, even Douyin—a platform that has just taken a hard regulatory stance—has no intention of outright banning AI-generated content. However, it cannot be denied that AIGC technology has significantly lowered the barrier to entry for content creation, shifting what was once considered an "artistic" process toward mechanical "technical" operations. As a result, numerous creators with varying levels of technical skill and compliance awareness have rushed into this space, chasing commercial gains while introducing serious legal and compliance risks.
The most pressing issue is fraud enabled through misuse of AI technology. According to relevant security reports, incidents of AI-based deepfake fraud surged by 3,000% in 2023 alone. Behind this alarming growth lies immense legal risk stemming from improper AI usage. Moreover, since AI-generated virtual characters rely on multiple AIGC components—including text, images, and audio—they remain vulnerable to intellectual property infringement issues prevalent across the AIGC landscape, particularly concerning character design, background music, and scriptwriting. Additionally, there are potential violations of individuals’ rights to likeness, privacy, and personal information. Using AIGC for misleading product promotion may also harm consumers' rights to informed decision-making and free choice.
In short, the application of AI virtual hosts in short videos and livestream commerce currently exists at the intersection of massive opportunity and significant risk.
Starting with Douyin: Compliance Analysis for AI-Generated Virtual Streamers
Therefore, it’s understandable why Douyin, as an industry leader, adopts a cautious approach toward regulating AI-generated content. In fact, this isn’t the first time Douyin has cooled enthusiasm around AI virtual hosts. As early as May 9, 2023, it issued the “Platform Norms and Industry Initiative on Artificial Intelligence Generated Content,” which clearly defined behavioral standards for AI-generated videos, images, and derivative virtual-person live streams. It also developed the “Watermark and Metadata Specification for AI-Generated Content” based on the Regulations on the Administration of Deep Synthesis Internet Information Services, further detailing requirements such as prominent labeling and preventing user confusion.
This latest “ban” primarily targets the persistent and expanding trend of using AI-generated virtual characters to publish content that defies scientific facts, fabricates falsehoods, spreads rumors, or deceives audiences. Specifically, three typical scenarios were highlighted:
First, creating fake foreigner personas with fabricated overseas identities to exploit patriotic sentiments and attract attention.
Second, generating idealized male or female avatars to deceive users into interactions, publishing emotionally manipulative content that simulates affection, redirecting users to private messaging apps, and even conducting scams.
Third, crafting false elite personas to disseminate low-quality inspirational quotes, pseudo-financial wisdom, counterfeit traditional culture teachings, Machiavellian tactics, or fake success philosophies—aimed at attracting followers cheaply and ultimately monetizing off-platform through paid courses or group memberships.
It is evident that all these cases involve using AI-generated virtual characters to target socially vulnerable groups with emotional needs, engaging in low-quality follower acquisition, traffic manipulation, and even off-platform fraud. In these schemes, the AI avatar itself is not always the direct instrument of deception; rather, it acts as a lure—a preliminary stage designed to appeal to specific audiences before progressing to more explicit illegal activities. This makes such misconduct particularly covert and dangerous.
Thus, what Douyin is restricting is not user creativity involving AI virtual hosts per se, but rather seeks to build a compliant ecosystem together with its users to minimize associated legal and regulatory risks. Viewing this move simply as a restriction on AI-based content creation does Douyin some injustice. The core objective remains to standardize risky behaviors and promote healthier development in the field through compliance measures.
User or Platform: Who Bears Responsibility for AI Virtual Host Compliance?
Douyin’s actions reflect its duty as both an internet service provider and a distributor of AI-generated content. Currently, China's regulations on generative AI services tend to place primary responsibility on platforms. For instance, the Basic Requirements for Generative Artificial Intelligence Service Security issued by the National Cybersecurity Standardization Technical Committee focuses evaluation and oversight responsibilities mainly on service providers. Similarly, Article 14 of the Interim Measures for the Administration of Generative Artificial Intelligence Services limits the so-called “safe harbor” principle, requiring providers to immediately rectify and report any illegal activities upon discovery. Article 15 mandates that service providers establish effective complaint and reporting mechanisms to ensure prompt handling of violations.
From this recent announcement, we can see Douyin reiterating these obligations. Beneath the stern tone lies a clear call for user cooperation—urging joint supervision and active reporting of inappropriate AI-generated content. At present, the key point of collaboration between platform and user centers on the first requirement: conspicuous labeling of AI-generated content. Per Article 12 of the Interim Measures, providers must label generated images, videos, and other content according to the Regulations on the Administration of Deep Synthesis Internet Information Services and the Cybersecurity Standards Practice Guide – Content Labeling Methods for Generative AI Services. Most social media platforms—including Douyin—have established clear rules regarding how such labels should be applied.
Currently, Douyin relies on users self-reporting AI-created content, after which the platform adds warning text. However, unless creators manually include tags like “virtual idol” or “AI avatar” in their descriptions, this system has notable enforcement gaps—especially when highly realistic AI-generated virtual figures are involved. While the new notice reinforces penalties for violators, given the current operational model, it appears more like an attempt to establish a tacit understanding between users and the platform: comply voluntarily, or face consequences. Though combining rule-building with detection algorithms seems to be the best available compliance strategy for service providers today, its effectiveness in this high-opportunity, high-risk environment remains to be proven in practice.
It should be emphasized that a virtual persona can never serve as a shield against accountability for its creator. Building compliance in AI creation is not something platforms can achieve unilaterally. What stands out most in Douyin’s latest notice—and underscores its strict stance and governance resolve—is its semi-transparent declaration following the list of violations: it will take internal disciplinary actions such as removing videos and suspending accounts for those misusing AI-generated virtual characters. Furthermore, for black-market groups already identified as using AI-generated avatars for criminal purposes, Douyin will file police reports and pursue legal action. Indeed, the kind of mutual understanding mentioned above should objectively exist between users and platforms. Creators engaging in rule-breaking behavior face consequences beyond mere platform sanctions—depending on severity, they may also face broader legal liabilities, including criminal prosecution.
Closing Thoughts
We often emphasize that regulation of new technologies and emerging fields doesn't necessarily hinder progress. Often, so-called “bans” also carry within them the intent to foster sustainable development through compliance. After all, only by walking the right path can one go far.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
