In April 2024, a company called Viggle AI powered a meme frenzy. For free, anyone could take a single still photo and swap themselves with a person in a video. The immediate hit was Lil Yachty Walks Out On Stage. A 2022 clip of Lil Yachty began morphing into the Joker, politicians, celebrities, and tech leaders.
Here’s my daughter (a dancer) going on stage based on a still image of her at a dance competition.
This is the single still image that I uploaded to Viggle. No need to mask it or anything. It guessed the back of the outfit.
Here’s a bit about how this works.
The critical prep for the Lil Yachty meme was done by a guy on Twitter with the handle AIWarper. He is the unsung hero of this one.
Since the Viggle template needs a clean reference video, AIWarper rotoscoped Lil Yachty using After Effects to create an isolated video of just Yachty.
How does one use Viggle?
Viggle is based in Discord, which is a sad gatekeeper for a lot of people who don’t understand how Discord works. It’s basically a simple chat-based command line interface where Viggle is a bot, and you can give it commands.
For the Lil Yachty meme, AIWarper loaded this into Viggle as a stored prompt.
From there, anyone can join Viggle on Discord and upload a still photo and call up the reference.
Discord commands usually start with / and with Viggle it’s /animate. After /animate you simply make four choices (below)
Image: is the image you want to animate.
MotionPrompt: $lil_yachty_stage_entrance (you have to find that by looking through the stored prompts, but it’s easy to spot since everyone’s using it)
Background: you use select “from template”. In this case, the choices are plain white background, a green screen, or the MotionPrompt template (aka the concert and the stage).
What is Viggle?
Viggle announced their new video swapping feature in March, and it looks a LOT like earlier tools from ByteDance.
Here’s the Viggle announcement:
ByteDance Work Based on TikTok Training
Here’s are some similar tools that preceded Viggle:
Text to Video
“ByteDance Introduces MagicVideo-V2: A Groundbreaking End-to-End Pipeline for High-Fidelity Video Generation from Textual Descriptions – MarkTechPost”
https://www.marktechpost.com/2024/01/16/bytedance-introduces-magicvideo-v2-a-groundbreaking-end-to-end-pipeline-for-high-fidelity-video-generation-from-textual-descriptions/
MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation – https://magicvideov2.github.io/
Another precursor was DiffPortrait 3D
Motion/Object Isolation (aka Segmentation)
Segmentation will eventually remove the need for video rotoscoping and masking.
Once the computer can track all objects in a video… it can replace all objects.
The Biggest Precursor- Magic Animates
Magic Animates – Bring Your Photos to Dance – https://magicanimates.com/
See how close this looks to segmentation (above)
And now… Viggle.. is just the beginning.
Viggle’s website is here – https://viggle.ai/ and it sends you to Discord.
Viggle says it’s powered by “JST-1, the first video-3d foundation model with actual physics understanding”. I’m about two weeks behind on my weekly newsletter, but I’ve never heard of JST-1… we shall see.
Matt Wolf has a great tutorial on how to use Viggle, on YouTube, if you want to try it yourself.
What does this mean?
Images: Stable diffusion got people addicted to text to image AI, and led to common usage of MidJourney and Dalle.
Chat: ChatGPT broke open chat bots.
Face Swapping: InsightFace (the API most tools use) brought face swapping into mainstream.
Video swapping: Viggle is a big deal.. because it starts to connect many themes: virtual reality, augmented reality, even Gaussian Splatting… with text and image based generative video… and also latent consistency models. Just as chat became multimodal (chatting about videos and images), generative AI just went multimodal as well.
In one year, we’ve gone from making silly images of otters on a plane, to face swapping, to full 3D video people swapping… using a single image.
I’d recommend following two trends: object segmentation and Gaussian splatting. Segmentation will all for tracking and swapping. Splatting will assist with the generation of smooth frames for the tough to render angles and elements that are not shown in a single photo.
I dedicated an entire newsletter to segmentation in December 2023.
https://ethanbholland.com/2023/12/24/ai-news-11-week-ending-12-22-2023-with-executive-summary-and-top-7-stories/
Here are a few more articles, if you’re interested in skimming about segmentation. I usually put any new links into the AR/VR category of this newsletter.
Leave a Reply