Startup accused of spreading racist audio deepfakes

ElevenLabs said it paused testing of its voice-cloning AI service due to “the growing number of abuse cases”. The report cites Motherboard.

Company representatives did not specify what violations were involved. However, journalists found that 4chan users used the product to create voices of celebrities to spread racist and homophobic statements.

In one clip, a voice impersonating actress Emma Watson reads an excerpt from “Mein Kampf”. In another deepfake, a political commentator Ben Shapiro makes racist remarks aimed at Congresswoman Alexandria Ocasio-Cortez.

The clips range from harmless to brutal, homophobic and racist. One of the compilations on 4chan contained a link to a beta version from ElevenLabs. Users suspected that this tool was used to create the clips.

The ElevenLabs website states that any registered user can use the service. To generate a voice, you must upload a sample longer than one minute.

ElevenLabs also offers “professional cloning”, which “can reproduce any accent”. Potential uses include voice-overs for information bulletins, books and videos, the startup’s website says.

The company was founded by former Google ML engineer Petr Dabkovski and former Palantir deployment strategist Mati Stanishevski. In January 2023 the startup announced that it had raised $2 million in a pre-seed round.

Soon after the clips circulated, ElevenLabs acknowledged the abuse problem.

Crazy weekend — thank you to everyone for trying out our Beta platform. While we see our tech being overwhelmingly applied to positive use, we also see an increasing number of voice cloning misuse cases. We want to reach out to Twitter community for thoughts and feedback!

— ElevenLabs (@elevenlabsio) January 30, 2023

“While we see that our technology being overwhelmingly used for good, we also observe an increase in cases of misuse of voice cloning,” the company said.

Representatives said they cannot trace any generated sound back to a specific user. However, the company is studying additional safety measures. These include requiring payment information, “full identity verification” to use the service, or manual review of each request.

In December 2022, Disney created an AI tool for rejuvenating actors.

In November, Intel unveiled a deepfake detector.

In September, scientists introduced a method for detecting audio deepfakes.

Subscribe to ForkLog news on Telegram: ForkLog AI — all the news from the world of AI!