Fakes, deepfakes and chatbots: how to spot AI manipulation online

Lies, deceit and manipulation have been with humanity since antiquity. With the rise of mass media, makers of fakes gained the ability to reach unprecedented audiences.

In the era of the rapidly expanding internet, disinformation has taken new forms. Websites, social networks, image boards and even traditional media have become venues for spreading false and manipulative claims. The risk may become acute in a new technological age—the age of artificial intelligence. Or does truth still have a chance to survive?

Bohdan Kaminsky explains how disinformation spreads, the role AI plays and what to expect next.

A brief history

Disinformation has long been a weapon in the hands of governments and intelligence services. The problem became especially acute during the cold war. The Soviet Union and the United States did not hesitate to manipulate data and interfere with information flows in order to blacken each other in the eyes of their own citizens.

The fall of the Berlin Wall might have eased tensions; it did not. Governments continued to distort the information space to suit their interests, using every tool at their disposal.

The end of the cold war coincided with the rapid spread of the internet, especially in Eastern Europe. As the machinery of censorship collapsed, a torrent of information washed over the former socialist bloc, making it ever harder for ordinary citizens to tell truth from falsehood.

The mid-2000s advent of social networks marked a new phase in the evolution of fakes and manipulation. The information space—and its audience—expanded so much that without careful fact-checking it became far harder to separate trustworthy reports from bunk.

New capabilities

Artificial intelligence developed alongside the internet, but its broad adoption in daily life came much later. One reason for the delay was high cost and the need for hefty computing power.

The situation changed dramatically in November 2022, when OpenAI released ChatGPT for public use. Transformer-based language models existed before, but this product, built on GPT-3, gave virtually any smartphone owner access to a modern chatbot.

Almost immediately, users began probing its limits. Fake posts, news items, songs, poems and other texts appeared online, spun up with AI. The service can generate vast volumes of prose in seconds on demand.

It did not stop at text. OpenAI also built the DALL-E series of image generators, capable of creating realistic pictures. Although Sam Altman’s tool has not yet been involved in disinformation scandals, rival platforms MidJourney and Stable Diffusion have.

In March 2023 the internet lit up with pictures of Pope Francis supposedly strutting the streets in a Balenciaga puffer. The images, it turned out, were generated by AI.

A pontiff in a fashion-forward coat may seem a harmless gag. Enthusiasts went further, conjuring images of former US president Donald Trump in a prison jumpsuit under police escort. This came just as he was expected to appear in court, so the fake photos drew wide notice.

Some users took the hyper-realistic frames at face value. The fake was quickly exposed, but the real impact of the disinformation campaign is hard to assess.

Video is different. Although today’s generators still struggle to produce realistic clips, deepfake technology has for years allowed one face to be swapped for another, or a person to be made to say words he never uttered.

One of the first high-profile deepfakes to spark heated discussion online was the 2017 programme Synthesizing Obama. It altered footage of former US president Barack Obama to make him appear to speak words taken from a separate audio track.

Since then the technology has been used for everything from porn deepfakes to information-psychological operations.

Criminals also forge audio to mimic other people’s voices. In 2021 fraudsters fabricated the speech of a large-company executive in the UAE and used it to steal $35m.

Synthetic voices have been used in disinformation campaigns as well. Early in 2024, in the thick of the US primary season, residents of New Hampshire получили a strange call from President Joe Biden. His synthesised voice urged voters not to back him in the upcoming primary.

The robocalls turned out to be a fake created by a political operative for Mr Biden’s intra-party rival to influence the result. The United States went on to ban AI robocalls, and the state prosecutor opened a criminal case over an attempted interference in the electoral process.

In a unanimous decision this week, the agency moved to make AI-generated voices in scam robocalls targeting consumers illegal. This would give State AGs across the country new tools to go after bad actors behind these nefarious calls.https://t.co/iFCOGsu7ue

— The FCC (@FCC) February 9, 2024

What helps today’s mischief-makers, who run AI-driven disinformation campaigns, is the easy availability of advanced tools. Chatbots churn out text; Stable Diffusion and MidJourney fake images; platforms like ElevenLabs create synthetic voices from short samples—all without deep technical skills.

How the industry is fighting AI fakes

The accessibility of modern AI tools has put the industry in a bind. Developers are criticised for shipping services that let almost anyone generate convincing content with minimal effort. Big players are therefore rolling out restrictions to deter manipulation.

OpenAI and Microsoft, the startup’s main investor, have installed filters that block generating content on certain topics. For example, ChatGPT will refuse to produce politically slanted or historically inaccurate material if the system suspects manipulative intent. DALL-E, for its part, will not generate images of public figures.

Those filters—perhaps others too—have made their way into Microsoft products: Bing, Copilot and Designer, all built on OpenAI models.

Google is also taking steps to combat manipulative and misleading content. At the end of 2022 the company выпустила guidance making clear that AI-generated text is treated as spam. The search engine and its algorithms prioritise high-quality text written by humans.

In 2023 the company also announced a tool to label AI images in search. How it will work is unclear, since many such images carry no watermarks or metadata to identify them. And if generators do add marks, they are easy to strip.

YouTube, owned by Google, has likewise declared war on AI content. In 2024 the company обязала creators to disclose if their work includes overt manipulation by artificial intelligence. Otherwise, the content faces removal. Again, it is unclear how YouTube plans to identify AI-laced videos if creators ignore the rule.

More broadly, tech giants set up the Coalition for Content Provenance and Authenticity (C2PA) back in 2021. Its aim is to add labels to AI content to help identify such photos and videos.

Meta has its own approach to AI content. After unveiling its Imagine generator, the company представила a labelling system that will watermark all of that tool’s outputs. Later Meta заявила it would add the label to AI images created by other models, too.

Much of the responsibility will rest on publishers, but the company did not rule out applying labels itself—label, not removal. Meta promised not to overuse its power to take down content that in one way or another violates rules on AI material.

ElevenLabs, mentioned earlier and used to create the Biden deepfake, also запретила cloning the voices of public figures. The platform’s rules already forbid using third-party data without consent, but the US president’s case showed the firm lacks adequate control.

MidJourney, the tool behind the Balenciaga Pope and Trump-in-custody memes, запретила prompts about politicians to stop users generating plausible fakes. The company’s head, David Holz, noted that he himself enjoys creating parodies of the former US president, but that it may be dangerous in terms of spreading disinformation.

Even so, as AI algorithms improve it will become harder for developers to curb manipulative content. Some tools, such as Stable Diffusion and Meta Llama 2, can already run locally on users’ devices and generate material without any restrictions.

Identifiers for AI content are therefore needed, but they are no panacea. OpenAI’s detector for AI-written text, for instance, failed at its task and was shut down. It more often labelled human writing as AI-generated than it correctly spotted AI text in the first place.

AI is itself a source of disinformation

You do not need a malicious actor to fall for manipulation and disinformation. Because of so-called hallucinations, AI can mislead users on its own.

In short, “language-model hallucinations” are the tendency of algorithms to “invent” information that does not in fact exist. Large language models do not think like humans. They are advanced predictive engines that look for patterns in text and try to generate the most plausible result.

Chatbots such as ChatGPT, Copilot, Gemini and Grok are prone to hallucinations. They make up statistics, facts and historical events.

A recent example is a news article, сгенерированная by the Grok chatbot from posts on X. It claims that the public is “alarmed by the Sun’s strange behaviour” and that “scientists cannot explain what is happening.” The reference was to the solar eclipse on 8 April 2024. Most likely, the chatbot took jokes on X—“куда пропало Солнце” (“where did the Sun go?”)—at face value; scientists, of course, understand eclipses perfectly well.

It is important to understand that a chatbot is not a search engine, and you should not take it at its word. If you use such tools for research, double-checking model outputs will not go amiss.

NVIDIA chief Jensen Huang offered a recommendation to help avoid AI hallucinations. He advised applying a journalist’s approach to verification:

• examine the source and its context;
• compare the source’s claims with established truths;
• if any part of the answer is untrue, discard the source and move on.

How to avoid falling for AI manipulation

If companies and detectors cannot yet reliably identify AI forgeries, does that mean all is lost? Hardly. As before, critical thinking remains the most reliable way to resist manipulation by AI-generated content.

Most fakes are debunked thanks to “human intelligence”. It is vital to treat any information on social media with scepticism. Saw the pope in a Balenciaga jacket? Check the Vatican’s official pages online.

Fact-checking has long been one of the main tools for debunking disinformation. It applies to AI manipulation, too. Fact-checking teams at major media outlets have increasingly заниматься verifying content for AI tampering.

A range of tools can also help check for AI involvement. For text, there are free tools such as GPTZero and Detecting-AI; for images, try Maybe’s AI Art Detector on Hugging Face. They are imperfect, but can offer a sense of whether AI fingerprints are present.

There is no shortage of manipulation and disinformation online even without artificial intelligence. Public figures are often credited with quotes they never uttered; photoshopped documents circulate, and so on.

AI will undoubtedly play a large role in increasing the volume of fakes online—if it is not doing so already. Yet the credulous are likely to fall for fakes anyway, as they did before the technology’s boom.

At least fact-checkers now have plenty more work.