Tearing Off the Virtual Human's "Business Face," DreamTalk Open-Sources Opportunities
TechFlow Selected TechFlow Selected
Tearing Off the Virtual Human's "Business Face," DreamTalk Open-Sources Opportunities
DreamTalk has been open-sourced, and this framework is expected to inject "emotion" into virtual humans.
Navigating Web3 tides with focused insightsBy Mu Mu
Riding the wave of large AI models, virtual digital humans—once a buzzword from the metaverse era—are making a comeback. Now equipped with artificial intelligence and natural language processing capabilities, these virtual beings, powered by 3D animation, motion capture, and human puppeteering ("zhong zhi ren"), finally have a "brain."
With AI integration, virtual humans that previously generated text and voice content can now interact more intelligently, delivering outputs richer in detail and expertise. More importantly, AI has significantly boosted productivity in their creation process.
In 2023, AI-enhanced virtual humans were widely adopted for content broadcasting and live streaming, becoming favorites among e-commerce brands and short-video creators.
Yet shortcomings remain. Human-like virtual characters still exhibit mechanical traits in facial expressions, voice, and movement, falling far short of real human authenticity. Some developers are turning to large AI models to bridge this gap, and DreamTalk is one such solution.
Developed jointly by Alibaba, Tsinghua University, and Huazhong University of Science and Technology, this diffusion-model-based framework starts from the "head"—enabling avatars not only to speak and sing but also to mimic facial expressions and synchronize lip movements.
Recently, DreamTalk was open-sourced—an advancement poised to infuse virtual humans with "emotion."
Enriching Emotional Expression in Virtual Humans
DreamTalk’s specialty lies in animating a static portrait—just one image—to make it talk, complete with synchronized lip movements and facial expressions, all driven by audio input.
The mimicking begins with "learning." Based on its large model architecture, DreamTalk learns expressive styles from input speech and portraits, generating personalized expression videos. Even using the same audio source, the avatar can display different emotional states—anger, joy, sadness, disdain—with corresponding facial cues.
DreamTalk supports diverse facial expressions
Solutions like DreamTalk add another powerful tool to virtual human production, addressing what is currently their biggest limitation: emotional expressiveness.
Joy lighting up the eyes, furrowed brows in anger, tears streaming down cheeks, radiant smiles—human emotions manifest instantly on the face, complemented by verbal expression. In contrast, most virtual humans wear a uniform "service smile," cycling between "smiling" and "cool pose" in their limited emoji repertoire. Despite various synthetic tones, their robotic nature remains obvious.
Earlier this year, Xiaoice launched its "GPT Clone Plan," integrating AI with virtual humans. The resulting AI clone influencer "Banzang Senlin" drew attention upon launch, but many criticized her stiff expressions and poor user experience.
Recently, Xiaoice's clones went live on Taobao flagship stores, yet facial rigidity persists. One buyer commented: "The video call feels utterly fake—has nothing to do with the original influencer."
A survey by the Communication University of China revealed clear differences in public expectations across virtual human types: over 60% of users prioritize appearance and work quality for virtual idols; 66% focus on hosting style for virtual streamers; and 50% value technical service and cross-industry collaboration abilities for virtual employees.
As virtual humans enter interactive spaces through live streaming, demand is shifting toward personalization. Especially with the rise of large AI models, solutions tackling virtual human emotion—and even sentiment—are emerging as an independent competitive frontier.
One of DreamTalk’s developers, Alibaba, previously filed a patent titled “Conversation Content Generation, Virtual Dialogue, and Data Processing Method.” According to the abstract, the method constructs an emotional association graph to deeply understand keywords triggering specific emotions in dialogue, enabling prediction of target keywords via emotional cues. This allows empathetic responses, achieving empathetic conversations and improving accuracy in content generation.
Entering 2024, market demands for virtual humans go beyond mere speech and motion—they must deliver full "emotional value."
AI Provides Not Just a Brain, But Emotion Too
AI solutions like DreamTalk, capable of matching facial expressions to voice and images, promise to enrich virtual humans’ expressive range while offering convenient tools for creators.
Technically, AI-powered simulation tools offer virtual humans the chance to break free from reliance on human operators.
Before the emergence of "algorithm-driven" virtual humans, most implementations relied on human-driven systems—what we commonly call "zhong zhi ren" (the person behind the avatar). These require real individuals using motion-capture equipment to record body movements, gaze, gestures, etc., before rendering. Typically, this approach involves longer production cycles and higher costs.
Liu Wei, founder of Hualian AI, once stated that only when virtual humans can be rapidly generated—solving the challenges of low-cost mass replication and high-frequency content output—while freeing themselves from human operators, will they achieve broad commercial viability.
Algorithm-driven virtual humans can autonomously learn lip movements, facial expressions, voice, posture, and gestures from training data without human involvement. They enable faster rendering and lower production costs, making this approach ideal for scalable virtual human creation.
From a user experience perspective, intelligent emotion generation will effectively enhance virtual humans’ interactive capabilities.
Numerous companion-type virtual human apps already exist. One app called Talkie, designed for virtual companionship, has achieved over a million daily active users overseas. Since August last year, its download numbers have remained consistently high, frequently ranking within the top ten entertainment apps on the U.S. Google Play Store. Beyond the U.S., Talkie has also performed well in developed markets such as New Zealand, the UK, Canada, and Australia.
Talkie’s virtual humans specialize in "chat companionship"
However, including Talkie itself, most current companion-focused virtual human apps offer limited emotional interactivity. Take Talkie: all virtual characters appear as static images against chat backgrounds, interacting solely through text—a pure "conversation companion."
As AI technology advances, apps like Talkie represent transitional forms rather than final destinations. Solutions addressing digital humans' emotional expression hold the potential to breathe a "soul" into virtual beings, marking a revolutionary opportunity for the next generation of virtual human products.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
