Home Learn An AI startup made a hyperrealistic deepfake of me that’s so good it’s scary

An AI startup made a hyperrealistic deepfake of me that’s so good it’s scary

An AI startup made a hyperrealistic deepfake of me that’s so good it’s scary

I’m stressed and running late, because what do you wear for the remainder of eternity? 

This makes it sound like I’m dying, nevertheless it’s the other. I’m, in a way, about to live eternally, due to the AI video startup Synthesia. For the past several years, the corporate has produced AI-generated avatars, but today it launches a brand new generation, its first to make the most of the most recent advancements in generative AI, and so they are more realistic and expressive than anything I’ve ever seen. While today’s release means almost anyone will now have the option to make a digital double, on this early April afternoon, before the technology goes public, they’ve agreed to make certainly one of me. 

After I finally arrive at the corporate’s stylish studio in East London, I’m greeted by Tosin Oshinyemi, the corporate’s production lead. He’s going to guide and direct me through the info collection process—and by “data collection,” I mean the capture of my facial expression, mannerisms, and more—very like he normally does for actors and Synthesia’s customers. 

He introduces me to a waiting stylist and a makeup artist, and I curse myself for wasting a lot time preparing. Their job is to make sure that people have the form of clothes that look good on camera and that they give the impression of being consistent from one shot to the subsequent. The stylist tells me my outfit is (phew), and the makeup artist touches up my face and tidies my baby hairs. The dressing room is decorated with lots of of smiling Polaroids of people that have been digitally cloned before me. 

Aside from the small supercomputer whirring within the corridor, which processes the info generated on the studio, this feels more like going right into a news studio than entering a deepfake factory. 

I joke that Oshinyemi has what might call a job title of the longer term: “deepfake creation director.” 

“We just like the term ‘synthetic media’ versus ‘deepfake,’” he says. 

Tosin Oshinyemi, the corporate’s production lead, guides and directs actors and customers through the info collection process.

It’s a subtle but, some would argue, notable difference in semantics. Each mean AI-generated videos or audio recordings of individuals doing or saying something that didn’t necessarily occur in real life. But deepfakes have a foul status. Since their inception nearly a decade ago, the term has come to signal something unethical, says Alexandru Voica, Synthesia’s head of corporate affairs and policy. Consider sexual content produced without consent, or political campaigns that spread disinformation or propaganda.

“Synthetic media is the more benign, productive version of that,” he argues. And Synthesia desires to offer one of the best version of that version.  

Until now, all AI-generated videos of individuals have tended to have some stiffness, glitchiness, or other unnatural elements that make them pretty easy to distinguish from reality. Because they’re so near the actual thing but , these videos could make people feel annoyed or uneasy or icky—a phenomenon commonly generally known as the uncanny valley. Synthesia claims its recent technology will finally lead us out of the valley. 

Due to rapid advancements in generative AI and a glut of coaching data created by human actors that has been fed into its AI model, Synthesia has been in a position to produce avatars which might be indeed more humanlike and more expressive than their predecessors. The digital clones are higher in a position to match their reactions and intonation to the sentiment of their scripts—acting more upbeat when talking about pleased things, as an illustration, and more serious or sad when talking about unpleasant things. In addition they do a greater job matching facial expressions—the tiny movements that may speak for us without words. 

But this technological progress also signals a much larger social and cultural shift. Increasingly, a lot of what we see on our screens is generated (or a minimum of tinkered with) by AI, and it’s becoming an increasing number of difficult to tell apart what’s real from what just isn’t. This threatens our trust in every thing we see, which could have very real, very dangerous consequences. 

“I believe we would just must say goodbye to checking out concerning the truth in a fast way,” says Sandra Wachter, a professor on the Oxford Web Institute, who researches the legal and ethical implications of AI. “The concept which you could just quickly Google something and know what’s fact and what’s fiction—I don’t think it really works like that anymore.” 

So while I used to be excited for Synthesia to make my digital double, I also wondered if the excellence between synthetic media and deepfakes is fundamentally meaningless. Even when the previous centers a creator’s intent and, critically, a subject’s consent, is there really a approach to make AI avatars safely if the top result is similar? And will we actually need to get out of the uncanny valley if it means we are able to not grasp the reality?

But more urgently, it was time to seek out out what it’s prefer to see a post-truth version of yourself.

Almost the actual thing

A month before my trip to the studio, I visited Synthesia CEO Victor Riparbelli at his office near Oxford Circus. As Riparbelli tells it, Synthesia’s origin story stems from his experiences exploring avant-garde, geeky techno music while growing up in Denmark. The web allowed him to download software and produce his own songs without buying expensive synthesizers. 

“I’m an enormous believer in giving people the power to precise themselves in the best way that they’ll, because I believe that that gives for a more meritocratic world,” he tells me. 

He saw the opportunity of doing something similar with video when he got here across research on using deep learning to transfer expressions from one human face to a different on screen. 

“What that showcased was the primary time a deep-learning network could produce video frames that looked and felt real,” he says. 

That research was conducted by Matthias Niessner, a professor on the Technical University of Munich, who cofounded Synthesia with Riparbelli in 2017, alongside University College London professor Lourdes Agapito and Steffen Tjerrild, whom Riparbelli had previously worked with on a cryptocurrency project. 

Initially the corporate built lip-synching and dubbing tools for the entertainment industry, nevertheless it found that the bar for this technology’s quality was very high and there wasn’t much demand for it. Synthesia modified direction in 2020 and launched its first generation of AI avatars for corporate clients. That pivot paid off. In 2023, Synthesia achieved unicorn status, meaning it was valued at over $1 billion—making it certainly one of the relatively few European AI firms to accomplish that. 

That first generation of avatars looked clunky, with looped movements and little variation. Subsequent iterations began looking more human, but they still struggled to say complicated words, and things were barely out of sync. 

The challenge is that folks are used to other people’s faces. “We as humans know what real humans do,” says Jonathan Starck, Synthesia’s CTO. Since infancy, “you’re really tuned in to people and faces. what’s right, so anything that’s not quite right really jumps out a mile.” 

These earlier AI-generated videos, like deepfakes more broadly, were made using generative adversarial networks, or GANs—an older technique for generating images and videos that uses two neural networks that play off each other. It was a laborious and complex process, and the technology was unstable. 

But within the generative AI boom of the last yr or so, the corporate has found it could actually create significantly better avatars using generative neural networks that produce higher quality more consistently. The more data these models are fed, the higher they learn. Synthesia uses each large language models and diffusion models to do that; the previous help the avatars react to the script, and the latter generate the pixels. 

Despite the leap in quality, the corporate remains to be not pitching itself to the entertainment industry. Synthesia continues to see itself as a platform for businesses. Its bet is that this: As people spend more time watching videos on YouTube and TikTok, there will likely be more demand for video content. Young individuals are already skipping traditional search and defaulting to TikTok for information presented in video form. Riparbelli argues that Synthesia’s tech could help firms convert their boring corporate comms and reports and training materials into content people will actually watch and interact with. He also suggests it may very well be used to make marketing materials. 

He claims Synthesia’s technology is utilized by 56% of the Fortune 100, with the overwhelming majority of those firms using it for internal communication. The corporate lists Zoom, Xerox, Microsoft, and Reuters as clients. Services start at $22 a month.

This, the corporate hopes, will likely be a less expensive and more efficient alternative to video from knowledgeable production company—and one which may be nearly indistinguishable from it. Riparbelli tells me its newest avatars could easily idiot an individual into considering they’re real. 

“I believe we’re 98% there,” he says. 

For higher or worse, I’m about to see it for myself. 

Don’t be garbage

In AI research, there’s a saying: Garbage in, garbage out. If the info that went into training an AI model is trash, that will likely be reflected within the outputs of the model. The more data points the AI model has captured of my facial movements, microexpressions, head tilts, blinks, shrugs, and hand waves, the more realistic the avatar will likely be. 

Back within the studio, I’m trying really hard to not be garbage. 

I’m standing in front of a green screen, and Oshinyemi guides me through the initial calibration process, where I even have to maneuver my head after which eyes in a circular motion. Apparently, it will allow the system to grasp my natural colours and facial expression. I’m then asked to say the sentence “All of the boys ate a fish,” which can capture all of the mouth movements needed to form vowels and consonants. We also film footage of me “idling” in silence.

image of Melissa standing on her mark in front of a green screen with server racks in background image
The more data points the AI system has on facial movements, microexpressions, head tilts, blinks, shrugs, and hand waves, the more realistic the avatar will likely be.

He then asks me to read a script for a fictitious YouTuber in several tones, directing me on the spectrum of emotions I should convey. First I’m purported to read it in a neutral, informative way, then in an encouraging way, an annoyed and complain-y way, and at last an excited, convincing way. 

“Hey, everyone—welcome back to along with your host, Jess Mars. It’s great to have you ever here. We’re about to tackle a subject that’s pretty delicate and truthfully hits near home—coping with criticism in our spiritual journey,” I read off the teleprompter, concurrently trying to visualise ranting about something to my partner through the complain-y version. “Regardless of where you look, it looks like there’s all the time a critical voice able to chime in, doesn’t it?” 

“That was really good. I used to be watching it and I used to be like, ‘Well, that is true. She’s definitely complaining,’” Oshinyemi says, encouragingly. Next time, possibly add some judgment, he suggests.   

We film several takes featuring different variations of the script. In some versions I’m allowed to maneuver my hands around. In others, Oshinyemi asks me to carry a metal pin between my fingers as I do. That is to check the “edges” of the technology’s capabilities relating to communicating with hands, Oshinyemi says. 

Historically, making AI avatars look natural and matching mouth movements to speech has been a really difficult challenge, says David Barber, a professor of machine learning at University College London who just isn’t involved in Synthesia’s work. That’s because the issue goes far beyond mouth movements; you’ve got to take into consideration eyebrows, all of the muscles within the face, shoulder shrugs, and the many different small movements that humans use to precise themselves. 

motion capture stage with detail of a mocap pattern inset
The motion capture process uses reference patterns to assist align footage captured from multiple angles across the subject.

Synthesia has worked with actors to coach its models since 2020, and their doubles make up the 225 stock avatars which might be available for patrons to animate with their very own scripts. But to coach its latest generation of avatars, Synthesia needed more data; it has spent the past yr working with around 1,000 skilled actors in London and Recent York. (Synthesia says it doesn’t sell the info it collects, even though it does release a few of it for educational research purposes.)

The actors previously got paid every time their avatar was used, but now the corporate pays them an up-front fee to coach the AI model. Synthesia uses their avatars for 3 years, at which point actors are asked in the event that they wish to renew their contracts. If that’s the case, they arrive into the studio to make a brand new avatar. If not, the corporate will delete their data. Synthesia’s enterprise customers may also generate their very own custom avatars by sending someone into the studio to do much of what I’m doing.

photograph of a teleprompter screen with three arrows pointing down to "HEAD then EYES>“‘ class=”wp-image-1091775″><figcaption class="wp-element-caption">The initial calibration process allows the system to grasp the topic’s natural colours and facial expression. </figcaption><div class="image-credit">DAVID VINTINER</div>
<div class="wp-block-column" style="width:;max-width:">
<div class="columns__content--3becc448c76a1a5a553df1358f1eebf3">
<div class="wp-block-image">
<figure class="wp-block-image size-large"><img decoding="async" src="https://wp.technologyreview.com/wp-content/uploads/2024/04/David_Vintiner__A7A0902.jpg?w=2588" alt="Melissa recording audio into a boom mic seated in front of a laptop stand" class="wp-image-1091781"><figcaption class="wp-element-caption">Synthesia also collects voice samples. Within the studio, I read a passage indicating that I explicitly consent to having my voice cloned.</figcaption><div class="image-credit">DAVID VINTINER</div>
<div class="">
<p class="imageSet__caption">
<div class="gutenbergContent__content--109b03a769a11e8ae3acbab352a64269 html_23">
<p>Between takes, the makeup artist is available in and does some touch-ups to be certain I look the identical in every shot. I can feel myself blushing due to the lights within the studio, but in addition due to the acting. After the team has collected all of the shots it must capture my facial expressions, I am going downstairs to read more text aloud for voice samples. </p>
<p>This process requires me to read a passage indicating that I explicitly consent to having my voice cloned, and that it could actually be used on Voica’s account on the Synthesia platform to generate videos and speech. </p>
<h3 class="wp-block-heading">Consent is vital</h3>
<p>This process may be very different from the best way many AI avatars, deepfakes, or synthetic media—whatever you should call them—are created. </p>
<p>Most deepfakes aren’t created in a studio. Studies have shown that the overwhelming majority of deepfakes online are nonconsensual sexual content, normally using images stolen from social media. Generative AI has made the creation of those deepfakes easy and low-cost, and there have been several high-profile cases within the US and Europe of kids and girls being abused in this fashion. Experts have also raised alarms that the technology will be used to spread political disinformation, a very acute threat given the record variety of elections happening world wide this yr. </p>
<div id="piano__post_body-desktop-4" class="piano__post_body"></div>
<p>Synthesia’s policy is to not create avatars of individuals without their explicit consent. But it surely hasn’t been immune from abuse. Last yr, researchers found pro-China misinformation that was created using Synthesia’s avatars and packaged as news, which the corporate said violated its terms of service. </p>
<p>Since then, the corporate has put more rigorous verification and content moderation systems in place. It applies a watermark with information on where and the way the AI avatar videos were created. Where it once had 4 in-house content moderators, people doing this work now make up 10% of its 300-person staff. It also hired an engineer to construct higher AI-powered content moderation systems. These filters help Synthesia vet each thing its customers attempt to generate. Anything suspicious or ambiguous, similar to content about cryptocurrencies or sexual health, gets forwarded to the human content moderators. Synthesia also keeps a record of all of the videos its system creates.</p>
<p>And while anyone can join the platform, many features aren’t available until people undergo an in depth vetting system much like that utilized by the banking industry, which incorporates talking to the sales team, signing legal contracts, and submitting to security auditing, says Voica. Entry-level customers are limited to producing strictly factual content, and only enterprise customers using custom avatars can generate content that incorporates opinions. On top of this, only accredited news organizations are allowed to create content on current affairs.</p>
<p>“We are able to’t claim to be perfect. If people report things to us, we take quick motion, [such as] banning or limiting individuals or organizations,” Voica says. But he believes these measures work as a deterrent, which implies most bad actors will turn to freely available open-source tools as an alternative. </p>
<div class="gutenbergContent__content--109b03a769a11e8ae3acbab352a64269 html_25">
<p>I put a few of these limits to the test after I head to Synthesia’s office for the subsequent step in my avatar generation process. With a view to create the videos that may feature my avatar, I even have to put in writing a script. Using Voica’s account, I resolve to make use of passages from in addition to previous articles I even have written. I also use a brand new feature on the Synthesia platform, which is an AI assistant that transforms any web link or document right into a ready-made script. I attempt to get my avatar to read news concerning the European Union’s recent sanctions against Iran. </p>
<p>Voica immediately texts me: “You bought me in trouble!” </p>
<div class="gutenbergContent__content--109b03a769a11e8ae3acbab352a64269 html_27">
<p>The system has flagged his account for attempting to generate content that’s restricted.</p>
<div class="wp-block-image">
<figure class="wp-block-image size-large"><img decoding="async" src="https://wp.technologyreview.com/wp-content/uploads/2024/04/WhatsApp-Image-2024-04-19-at-10.29.54.jpg?w=1280" alt=
AI-powered content filters help Synthesia vet each thing its customers attempt to generate. Only accredited news organizations are allowed to create content on current affairs.

Offering services without these restrictions can be “a fantastic growth strategy,” Riparbelli grumbles. But “ultimately, we have now very strict rules on what you possibly can create and what you can’t create. We predict the appropriate approach to roll out these technologies in society is to be just a little bit over-restrictive in the beginning.” 

Still, even when these guardrails operated perfectly, the last word result would nevertheless be an online where every thing is fake. And my experiment makes me wonder how we could possibly prepare ourselves. 

Our information landscape already feels very murky. On the one hand, there’s heightened public awareness that AI-generated content is flourishing and may very well be a robust tool for misinformation. But on the opposite, it remains to be unclear whether deepfakes are used for misinformation at scale and whether or not they’re broadly moving the needle to vary people’s beliefs and behaviors. 

If people change into too skeptical concerning the content they see, they may stop believing in anything in any respect, which could enable bad actors to make the most of this trust vacuum and lie concerning the authenticity of real content. Researchers have called this the “liar’s dividend.” They warn that politicians, for instance, could claim that genuinely incriminating information was fake or created using AI. 

Claire Leibowicz, the top of the AI and media integrity on the nonprofit Partnership on AI, says she worries that growing awareness of this gap will make it easier to “plausibly deny and solid doubt on real material or media as evidence in many various contexts, not only within the news, [but] also within the courts, within the financial services industry, and in lots of our institutions.” She tells me she’s heartened by the resources Synthesia has dedicated to content moderation and consent but says that process is rarely flawless.

Even Riparbelli admits that within the short term, the proliferation of AI-generated content will probably cause trouble. While people have been trained to not imagine every thing they read, they still are likely to trust images and videos, he adds. He says people now must test AI products for themselves to see what is feasible, and shouldn’t trust anything they see online unless they’ve verified it. 

Never mind that AI regulation remains to be patchy, and the tech sector’s efforts to confirm content provenance are still of their early stages. Can consumers, with their various degrees of media literacy, really fight the growing wave of harmful AI-generated content through individual motion? 

Be careful, PowerPoint

The day after my final visit, Voica emails me the videos with my avatar. When the primary one starts playing, I’m shocked. It’s as painful as seeing yourself on camera or hearing a recording of your voice. Then I catch myself. At first I assumed the avatar me. 

The more I watch videos of “myself,” the more I spiral. Do I actually squint that much? Blink that much? And move my jaw like that?

It’s good. It’s really good. But it surely’s not perfect. “Weirdly good animation,” my partner texts me. 

“However the voice sometimes sounds exactly such as you, and at other times like a generic American and with a weird tone,” he adds. “Weird AF.” 

On this AI-generated footage, synthetic “Melissa” gives a performance of Hamlet’s famous soliloquy.

He’s right. The voice is typically me, but in real life I and more. What’s remarkable is that it picked up on an irregularity in the best way I talk. My accent is a transatlantic mess, confused by years spent living within the UK, watching American TV, and attending international school. My avatar sometimes says the word “robot” in a British accent and other times in an American accent. It’s something that probably no one else would notice. However the AI did. 

My avatar’s range of emotions can be limited. It delivers Shakespeare’s “To be or to not be” speech very matter-of-factly. I had guided it to be furious when reading a story I wrote about Taylor Swift’s nonconsensual nude deepfakes; the avatar is complain-y and judgy, obviously, but not offended. 

This isn’t the primary time I’ve made myself a test subject for brand spanking new AI. Not too way back, I attempted generating AI avatar images of myself, only to get a bunch of nudes. That have was a jarring example of just how biased AI systems will be. But this experience—and this particular way of being immortalized—was definitely on a distinct level.

Carl Öhman, an assistant professor at Uppsala University who has studied digital stays and is the creator of a brand new book, , calls avatars just like the ones I made “digital corpses.” 

“It looks exactly such as you, but nobody’s home,” he says. “It will be the equivalent of cloning you, but your clone is dead. And then you definately’re animating the corpse, in order that it moves and talks, with electrical impulses.” 

That’s form of the way it feels. The little, nuanced ways I don’t recognize myself are enough to place me off. Nevertheless, the avatar could quite possibly idiot anyone who doesn’t know me thoroughly. It really shines when presenting a story I wrote about how the sector of robotics may very well be getting its own ChatGPT moment; the virtual AI assistant summarizes the long read into a good short video, which my avatar narrates. It just isn’t Shakespeare, nevertheless it’s higher than lots of the corporate presentations I’ve had to take a seat through. I believe if I were using this to deliver an end-of-year report back to my colleagues, possibly that level of authenticity can be enough. 

And is the sell, in keeping with Riparbelli: “What we’re doing is more like PowerPoint than it’s like Hollywood.”

Once a likeness has been generated, Synthesia is in a position to generate video presentations quickly from a script.

The latest generation of avatars actually aren’t ready for the silver screen. They’re still stuck in portrait mode, only showing the avatar front-facing and from the waist up. But within the not-too-distant future, Riparbelli says, the corporate hopes to create avatars that may communicate with their hands and have conversations with each other. Additionally it is planning for full-body avatars that may walk and move around in an area that an individual has generated. (The rig to enable this technology already exists; the truth is it’s where I’m within the image at the highest of this piece.)

But will we want that? It looks like a bleak future where humans are consuming AI-generated content presented to them by AI-generated avatars and using AI to repackage that into more content, which can likely be scraped to generate more AI. If nothing else, this experiment made clear to me that the technology sector urgently must step up its content moderation practices and make sure that content provenance techniques similar to watermarking are robust. 

Even when Synthesia’s technology and content moderation aren’t yet perfect, they’re significantly higher than anything I even have seen in the sector before, and that is after only a yr or so of the present boom in generative AI. AI development moves at breakneck speed, and it’s each exciting and daunting to think about what AI avatars will seem like in only a number of years. Possibly in the longer term we may have to adopt safewords to point that you simply are the truth is communicating with an actual human, not an AI. 

But that day just isn’t today. 

I discovered it weirdly comforting that in certainly one of the videos, my avatar rants about nonconsensual deepfakes and says, in a sociopathically pleased voice, “The tech giants? Oh! They’re making a killing!” 

I’d never. 


Please enter your comment!
Please enter your name here