Civitai, a hugely popular site for sharing AI models and a critical piece of infrastructure for AI-generated pornography across the internet, yesterday announced that it will now ban content depicting incest, self-harm, and fetishes featuring “bodily excretions” like urine, vomit, menstruation, and diapers. The site is also demonetizing content featuring the likeness of real people and making it harder to find.
“As many of you know, the world of AI content is under increasing scrutiny – from regulators, legislators, and the general public,” Civitai said in a post explaining the policy and content “adjustments.” “Platforms like ours are being held to higher standards around safety, legality, and accountability. We’ve always striven to be a place for innovation, creativity, and openness, but some things have to change in order to continue operating, growing, and protecting our users.” Depictions of firearms being pointed at people and “mind-altered states” like hypnosis are also now forbidden on Civitai.
Despite its claim that it’s being held to a higher standard, the type of content Civitai has updated its policies to ban has long been on the edge of what’s acceptable even on porn sites, and is in fact banned on many platforms. While it’s an established fetish that consenting adults engage with regularly, adult content involving diapers is especially sensitive for internet platforms because it could be interpreted as featuring minors. In 2023, for example, Sam reported that Patreon had banned a ton of accounts that created “adult baby/diaper lover,” or ABDL content which was popular in the furry community and a source of income that suddenly disappeared for some.
Incest as well is normally a step too far for most porn sites because it’s illegal in many jurisdictions, which is why so many adult videos engage with this taboo via the now common porn trope of step- sister/brother/mother/father. Weapons, content depicting self harm or harm to others, and content featuring certain bodily fluids are also commonly prohibited even on adult platforms. All of these are banned on OnlyFans, for example.
Civitai is giving users 30 days before it starts removing content violating these new policies. Click any of these links at your own risk, but a quick search reveals that Civitai currently does hold many AI image generation models specifically for the purpose of generating images of people inserting guns into vaginas, ABDL, menstrual pads, and so on. Did I mention this website is backed by Andreessen Horowitz, one of the biggest venture capital firms in silicon valley? Anyway…
On one hand, despite trying to downplay the obvious popularity of adult content on its site and its critical role in making it easier for people to generate non-consensual adult content, in many ways the policy update shows that Civitai is dealing with the same moderation challenges that all adult sites deal with, and that it’s subject to the same pressures. One of the big reasons adult sites and other internet platforms are banning the same type of content is that they are all getting pressured by payment processors, without which it is much more difficult for them to make money. This is famously why Pornhub changed its entire approach to content moderation and banned millions of unverified content from its platform in 2020. Civitai, Visa, and Mastercard did not respond to requests for comment.
On the other hand, Civitai’s moderation might be more challenging because AI image generators make it incredibly easy to produce an infinite stream of images that will create more or less whatever a user describes in a prompt. It’s just a massive amount of wild content.
That’s probably why Civitai also announced that it now demands users include generation metadata on all mature content, and why it announced it has partnered with AI moderation service Clavata, “whose image ingestion and analysis system represents the most advanced and accurate solution we have encountered to-date,” according to Civitai.
“The one upside of the changes will be not seeing 20 new diaper loras [AI models] a day,” one user said in the official Civitai Discord. “But the new censorship is a bit over the line when it gets into common themes like drunk hanky panky and gunplay.”
“Even though scat, diapers, piss, etc are squicky for me, I’m always apprehensive when they get censored, because then it’s usually a matter of time until things that I do enjoy, and other things, get censored as well,” another user said. “The canary in the cave, as it were.”
Other updates to Civitai’s policy signal that the site is finally taking some type action on models that are designed to recreate the likeness of real people. Content tagged with the names of real celebrities or “real-person” will be hidden from feeds, ads will not show up next to this content, and users will not be able to tip each other with Buzz, Civitai’s on-platform currency, for this content. The latter is especially notable because Civitai had previously introduced a “bounties” feature which allowed people to offer a certain amount of Buzz to any other user who created an AI model for generating the likeness of real people.
This isn’t the first time Civitai has done a major overhaul of its content moderation policies. Following 404 Media’s investigation in 2023 which showed that Civitai users were generating images that “could be categorized as child pornography,” the platform introduced radical new measures to stop abuse on its site.
