Sex and Politics: Deepfakes Make a Big Splash to Kick Off 2024
Joe Biden and Taylor Swift Deepfakes, technology, and society
Deepfakes are on the rise. Scams featuring Tom Hanks, MrBeast, Elon Musk, and consumer advocate Martin Lewis made headlines in 2023. These scams were about money. Others were motivated by a mix of humor, trolling, or misinformation. This past week, we saw two examples of manipulation that have raised public concern to new levels and spotlighted two other areas of concern: sex and politics
Taylor Swift Targeted
Last week, sexually explicit deepfake images of Taylor Swift began appearing on social media. The Verge reported that:
[A post] on X attracted more than 45 million views, 24,000 reposts, and hundreds of thousands of likes and bookmarks before the verified user who shared the images had their account suspended for violating platform policy. The post was live on the platform for around 17 hours prior to its removal.
404Media reported that the images first appeared on 4Chan and then migrated to a Telegram group and X. The outlet speculates that the images may have been made using Microsoft Designer and the text-to-image generator added to the product in 2023.
The Telegram group recommends that members use Microsoft’s AI image generator called Designer, and users often share prompts to help others circumvent the protections Microsoft has put in place. The 4chan thread where these images appeared also included instructions on how to make Microsoft's Designer make explicit images. For example, 404 Media’s testing found that Designer will not generate an image of “Jennifer Aniston,” but we were able to generate suggestive images of the actress by using the phrase “ jennifer ‘actor’ aniston.” Prior to the Swift AI images going viral on Twitter, a user in the Telegram group recommended that members use the phrase “Taylor ‘singer’ Swift” to generate images. 404 Media was unable to recreate the type of images that were posted to Twitter, but we found that Microsoft’s Designer would not generate images of “Taylor Swift,” but did generate images of “Taylor ‘singer’ Swift.”
This led to media outlets asking Microsoft about the risks and what steps are taken to prevent these types of incidents. Microsoft told 404 Media on Friday that it was investigating the situation and confirmed on Monday that updates were made to strengthen the image generation safety filters.
Microsoft has introduced more protections to Designer, an AI text-to-image generation tool that people were using to make nonconsensual sexual images of celebrities…Microsoft said that an ongoing investigation was unable to verify that the images of Swift on Twitter were made with Designer, but that it continues to strengthen its text filtering prompts and address misuse of its services.
Joe Biden Robocaller
The story behind a Joe Biden voice clone that robocalled New Hampshire voters also has unconfirmed origins, but AI researchers at Pindrop Security have identified the technology used in the scam. A news release from the New Hampshire Attorney General’s office said:
The Attorney General’s Office has received complaints regarding a recorded message encouraging voters not to vote in the January 23, 2024, New Hampshire Presidential Primary Election. The message, which was sent on January 21, 2024, stated “your vote makes a difference in November, not this Tuesday.” Although the voice in the robocall sounds like the voice of President Biden, this message appears to be artificially generated based on initial indications…These messages appear to be an unlawful attempt to disrupt the New Hampshire Presidential Primary Election and to suppress New Hampshire voters.
Vijay Balasubramaniyan, CEO and co-founder of Pindrop, wrote in a Thursday blog post about the process used to identify voice clone sources.
In a groundbreaking development within the 2024 US election cycle, a robocall imitating President Joe Biden was circulated. Several news outlets arrived at the right conclusion that this was an AI-generated audio deepfake that targeted multiple individuals across several US states. However, many mentioned how hard it is to identify the TTS engine used (“It’s nearly impossible to pin down which AI program would have created the audio” – NBC News). This is the challenge we focussed on, and our deep fake analysis suggests that the specific TTS system used was ElevenLabs.
Pindrop’s deepfake engine analyzed the 39-second audio clip through a four-stage process: audio filtering & cleansing, feature extraction, breaking the audio into 155 segments of 250 milliseconds each, and continuous scoring all the 155 segments of the audio…Using our proprietary deepfake detection engine, we assigned ‘liveness’ scores to each segment, ranging from 0 (synthetic) to 1.0 (authentic). The liveness scores of this Biden robocall consistently indicated an artificial voice. The score fell below the liveness threshold of 0.3 after the first 2 seconds and stayed there for the rest of the call, clearly identifying it as a deepfake…
Pindrop’s deepfake detection engine found, with a 99% likelihood, that this deepfake is created using ElevenLabs or a TTS system using similar components. We ensured that this result doesn’t have an overfitting or a bias problem by following research best practices. Once we narrowed down the TTS system used here to ElevenLabs, we then validated it using the ElevenLabs SpeechAI Classifier, and we obtained the result that it is likely that this audio file was generated with ElevenLabs (84% likely probability).
What it Means
Voicebot and Pindrop conducted a U.S. consumer survey in mid-2023 to gauge consumer exposure and interest in deepfakes and voice clones. Reactions to the technology were mixed. Consumers were most likely to view the technology either “very favorably” or “very unfavorably,” with a moderate skew toward unfavorability. However, over 90% of consumers expressed at least some concern about deepfakes and voice clones, with about 60% being “highly” or “extremely” concerned.
Situations that involve high-profile celebrities and politicians may increase those concerns. There is a lot of discussion about using visual and auditory watermarks to identify AI-generated content. The thinking is that this will increase consumer trust in digital content. However, the scam artists, fraudsters, and trolls aren’t going to voluntarily tag their content as a deepfake. It undermines their objective. Implementing deepfake and voice clone detection solutions is the most practical method to address the problem. It is the only way to quickly and reliably identify nefarious deepfakes.
I am not arguing against watermarking. If people want to add that badge to their content, it may provide some value, particularly for use in entertainment where fiction and frivolity are embedded in the art. With that said, there is a potential downside to this as well. If most organizations and independent creators dutifully label their AI-generated work, making it common across the web, it may increase the subconscious sentiment that unlabeled content is real. That would play into the hands of the scammers’ intent to deceive.
So, label content all you like. Detection is all you need. Fast. Reliable. Pervasive.
The problem with detection is that it only works after the fact. In many cases, the damage has already been done.
Today I read in a LinkedIn post the following story: “My "brother" called our mother saying he's been in a car crash and is being charged with criminal negligence and needs $8500 immediately. They spoke for a few minutes and my mother said his manurisms were spot on. She recalled my warning about scams like this happening, and asked him questions only my real brother would know. They immediately hung up. She then called my brother who has, of course, been at work all day.”
I have seen websites where all you need to do is select a voice or create a custom voice, type what you want the AI to say, and press dial.
Honestly, I see no real solution to this, other than that we have to become more vigilant ourselves. We, as people, need to become better AI detectors.